Systems and methods for providing a dynamic continual improvement educational environment

ABSTRACT

Dynamic systems and methods for gathering/tracking data, automatically adapting to an individual&#39;s pace of learning, selectively determining the type of content provided to an individual, selectively providing an exposure frequency for the content, and/or enabling rapid design modifications within the educational environment. Educational content is dynamically designed and customizably presented to an individual learner. An analysis is performed on the data to optimize learning. Modifications are selectively or automatically made to the educational content. The process of designing, implementing, analyzing, and selectively modifying creates a cycle that optimizes the learning process and adapts to individual learners. Furthermore, the educational content is dynamically provided to the learner on an iterative basis according to the need of that learner in the learning process.

RELATED APPLICATIONS

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 10/632,892, filed Jul. 31, 2003, entitled SYSTEMSAND METHODS FOR PROVIDING A DYNAMIC CONTINUAL IMPROVEMENT EDUCATIONALENVIRONMENT, which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to providing a dynamic continualimprovement educational environment. In particular, the presentinvention relates to dynamic systems and methods for gathering/trackingdata, automatically adapting to an individual's pace of learning,selectively determining the type of content provided to an individual,selectively providing an exposure frequency for the content, and/orenabling rapid design modifications within the educational environment.

2. Background and Related Art

Historically, a variety of techniques have been utilized to teacheducational concepts with varying degrees of success. Some techniquesinclude instructor-led classrooms, wherein a group of students areexposed to lessons given by the instructor. While this technique isavailable, a typical ratio of instructor per number of students limitsthe amount of individualized training that can occur with the students.

The emergence of the personal computer has allowed for electronicteaching techniques to be used, including a technique that allowsmultimedia to be used. Enhanced presentations are used in classroomenvironments to provide exciting/captivating educational lessons. Whilethis technique may yield more interesting lessons, the amount ofindividualized training is not increased.

Computers have been utilized in other teaching techniques, such ascomputer based training (CBT) or computer based instruction (CBI). Thesecomputer based techniques allow a student to interface with a computerprogram having instructional content rather than an instructor to enablethe classes to be available at the convenience of the student. Thus,while instructor-led classes can become full, computer based classes arealways available.

In some computer based training, a series of static electronic lessonsare provided that are separated by a prompt-response testing procedure.The testing procedure determines whether the student is allowed toprogress to the next lesson or is alternatively required to return toadditional instruction on the tested subject matter to better understandthe material. These techniques monitor student progress and disseminateadditional information as the student progresses.

Other educational techniques employ computer technology, but are limitedin scope to particular fields of instruction (e.g., instruction on theuse of computer programs) or are limited in format to specific media(e.g., text and simulation exercises). Other techniques use lessons orobjectives arranged in a predefined hierarchy, or focus on monitoringand evaluating the student rather than on providing instruction to thestudent.

Many current educational techniques utilize a static lesson format thatis typically arranged in a predefined order. The lesson format isirrelevant to the individual needs of each student and requiresconformity to a static learning method that may not fit a student'sspecific learning needs.

Thus, while educational techniques are available for use in teachingeducational content, challenges still exist with the current techniques.For example, current techniques are typically static in nature, are timeconsuming in their creation, and the underlining principles ofparticular techniques are not strictly followed, preventing accurateanalysis of their implementation. Accordingly, it would be animprovement in the art to augment or even replace current techniqueswith other techniques.

SUMMARY OF THE INVENTION

The present invention relates to providing a dynamic continualimprovement educational environment. In particular, the presentinvention relates to dynamic systems and methods for gathering/trackingdata, automatically adapting to a characteristic of an individual (e.g.,the individual's pace, background, style, and/or progress in learning),selectively determining the type and difficulty of content provided toan individual, selectively providing an exposure frequency for thecontent, and/or enabling rapid design modifications within theeducational environment.

Implementations of the present invention take place in association witha dynamic learning process that includes the ability to design ordevelop an educational experience or otherwise provide educationallessons (e.g., educational activities, educational content, etc.). Thedesigning of the educational experience or lesson is facilitated byutilizing an object oriented format, a drag-and-drop interface, or otherprocess that facilitates design development of educational content anddoes not require a computer programmer to develop the educationalexperience. Once designed, the implementation of the educational lessonis experienced by an individual, sometimes in collaboration with a peeror tutor, and includes providing instruction and gathering data. Ananalysis is performed on the data to optimize learning. The analysiscorresponds to a particular individual, group and/or educational lesson.Modifications are selectively or automatically made to the educationallesson. The process of designing, implementing, analyzing, andselectively modifying creates a cycle that optimizes the learningprocess and adapts to groups and individual learners with the goal ofimproving learning outcomes and efficiency.

At least some implementations of the present invention embrace theutilization of a computer device in designing and developing educationallessons, implementing educational lessons, performing analysis and/orproviding modifications. An individual learner/student may interfacewith the computer device in the educational environment and mayadditionally interface with an instructor. If the instructor (or a peer)is at a distant location, their collaboration may be mediated throughmultiple computers connected to the internet or some other network.

Within the environment, the educational lesson or content is dynamicallyprovided to the learner on an iterative basis according to the need ofthat learner in the learning process. Learner performance data isgathered and is selectively used to adjust the pace of learning, tomodify the frequency of exposure to particular content, and to regulatethe type and difficulty of content to which the learner is exposed.

Aspects of the educational environment are easily and/or automaticallyadapted to a learner's performance. For example, if an analysis of userdata indicates that a given learning activity needs an additionalfeature, that feature can quickly be added or created, such as throughthe use of a drag-and-drop interface. Similarly, if a given activityproves to be unhelpful, it can be immediately eliminated. Also, if aparticular activity proves to be helpful to some learners, but not toothers, entry conditions are set to only allow those learners that arepredicted to benefit from the activity to be exposed to the activity.Factors or characteristics of the user that may be taken into accountinclude age, native language, learning style, institution, background,interests, purpose in learning, degree of long-term retention desired,the breadth or depth of their overall mastery, and their need to beparticularly well-prepared to use a certain subset of the informationfor an upcoming responsibility or engagement (such as an academicconference on a particular subject), etc.

Other learners skip or never experience the particular activity.Alternatively, they may be exposed to another educational activity. If anew activity would be useful to the learner, the new activity may bequickly provided or otherwise authored by a designer rather than aprogrammer.

Automatic or partially automated studies determine the effectiveness ofparticular lessons, activities, or other instructional design decisions.For example, an assignment may be made for a first test group toexperience a first lesson and a second test group to experience a secondlesson. In another example, one learning activity might be presented ina different way to one group than another group. In both examples, thegroups are formed automatically by random assignment of treatment tosubjects. The results of the two test groups are analyzed to determinethe effectiveness of the two lessons in relation to each other.

The methods and processes of the present invention have proven to beparticularly useful in the area of teaching a foreign language to thelearner. However, those skilled in the art will appreciate that themethods and processes of the present invention can be used in a varietyof different applications and in a variety of different educationalenvironments to teach any type of educational topic or material. Forexample the educational content may embrace language, mathematics,science, technical training, cooking, medical procedures, a particularskill, professional training, or any other learning.

These and other features and advantages of the present invention will beset forth or will become more fully apparent in the description thatfollows and in the appended claims. The features and advantages may berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. Furthermore, thefeatures and advantages of the invention may be learned by the practiceof the invention or will be obvious from the description, as set forthhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the manner in which the above recited and other featuresand advantages of the present invention are obtained, a more particulardescription of the invention will be rendered by reference to specificembodiments thereof, which are illustrated in the appended drawings.Understanding that the drawings depict only typical embodiments of thepresent invention and are not, therefore, to be considered as limitingthe scope of the invention, the present invention will be described andexplained with additional specificity and detail through the use of theaccompanying drawings in which:

FIG. 1 illustrates a representative system that provides a suitableoperating environment for use of the present invention;

FIG. 2 illustrates a representative networked system for use inassociation with an embodiment of the present invention;

FIG. 3 illustrates a representative relationship within a continualimprovement educational process;

FIGS. 4-5 illustrate a representative user interface in association withan activity builder tool;

FIG. 6 illustrates a representative user interface in association withan adaptive path builder tool;

FIGS. 7-8 illustrate a representative user interface in association witha theme designer tool;

FIG. 9 illustrates a representative user interface in association withan audio cutter tool;

FIG. 10 illustrates a representative user interface for use inassociation with a concept type definer tool;

FIG. 11 illustrates a representative user interface in association witha concept entry tool;

FIGS. 12-13 illustrate a representative user interface for associationwith a content checker tool;

FIG. 14 illustrates a representative user interface for association witha tagger tool;

FIG. 15 illustrates a representative user interface in association witha gloss linker tool;

FIG. 16 illustrates a representative user interface for association witha media manager tool;

FIG. 17 illustrates a representative user interface for association witha relationship linker tool;

FIG. 18 illustrates a representative user interface for association witha sentence synchronizer;

FIGS. 19-20 illustrate a representative user interface for associationwith a text importer tool;

FIGS. 21-23 illustrate a representative user interface for associationwith a translation editor tool;

FIG. 24 illustrates a representative user interface in association witha component tester tool;

FIGS. 25-27 illustrate a representative user interface in associationwith an instruction engine;

FIG. 28 illustrates a representative user interface in association witha user manager tool;

FIGS. 29-30 illustrate a representative user interface in associationwith an object manager tool;

FIG. 31 illustrates a representative user interface in association witha group manager tool;

FIG. 32 illustrates a block diagram in association with a storyboard ofremote document feedback program;

FIG. 33 illustrates a representative user interface in association witha rights manager tool;

FIG. 34 illustrates a block diagram and provides a representativeassociation between a builder and a reporting system;

FIG. 35 illustrates a representative user interface having an editscreen associated with a reports builder tool;

FIG. 36 illustrates a representative user interface in association witha frequency screen of a reports builder tool;

FIG. 37 illustrates another representative user interface in associationwith a frequency screen of a report builder tool;

FIG. 38 illustrates another representative user interface in associationwith a frequency screen of a reports builder tool;

FIG. 39 illustrates a representative user interface relating to a groupscreen in association with a reports builder tool;

FIG. 40 illustrates another representative user interface relating to agroup screen in association with a reports builder tool;

FIG. 41 illustrates a representative user interface in association withan edit screen relating to a reports builder tool, wherein the editscreen is on a question level;

FIG. 42 illustrates a representative user interface in association withan edit screen relating to a reports builder tool, wherein the editscreen is on an item level;

FIG. 43 illustrates a representative user interface in association witha reports builder tool;

FIGS. 44-45 illustrate a representative user interface in associationwith a tutor guidance system; and

FIG. 46 illustrates a representative user interface in association witha research organizer tool.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to providing a dynamic continualimprovement educational environment. In particular, the presentinvention relates to dynamic systems and methods for gathering/trackingdata, automatically adapting to information about an individual (e.g.,the individual's pace of learning, background, style of learning,progress in learning, etc.) selectively determining the type anddifficulty of content provided to an individual, selectively providingan exposure frequency for the content, and/or enabling rapid designmodifications within the educational environment.

In the disclosure and in the claims the term “design” or “designing”shall refer to any phase in creating, developing, organizing or craftingeducational content for presentation to one or more users, includingcrafting how the educational content shall function, generating theinstructional system or educational content for presentation, or thelike.

Embodiments of the present invention take place in association with adynamic learning process that includes the ability to design or developan educational experience or otherwise provide educational lessons(e.g., educational activities, educational content, etc.). The designingof the educational experience or lesson is facilitated by utilizing anobject oriented format, a drag-and-drop interface, or another designprocess that eases development. Once designed, the educational lesson isexperienced by an individual, includes instruction and gathers data. Ananalysis is performed on the data to optimize learning, and maycorrespond to a particular individual, group and/or educational lesson.Modifications are selectively or automatically made to the educationallesson to enhance learning. The process of designing, implementing,analyzing, and selectively modifying creates a cycle that optimizes thelearning process and adapts to groups and individual learners with thegoal of improving learning outcomes and efficiency.

While the methods and processes of the present invention have proven tobe particularly useful in the area of teaching a foreign language to thelearner, and will be described in terms of teaching a foreign language,those skilled in the art will appreciate that the methods and processesof the present invention embrace a variety of different applicationsand/or a variety of different educational environments to teach any typeof educational topic or material. For example the educational contentmay embrace language, mathematics, science, cooking, technical training,medical procedures, a particular skill, professional training, or anyother educational learning.

The following disclosure of the present invention is grouped into twosubheadings, namely “Exemplary Operating Environment” and “DynamicContinual Improvement Educational Environment.” The utilization of thesubheadings is for convenience of the reader only and is not to beconstrued as limiting in any sense.

Exemplary Operating Environment

At least some embodiments of the present invention embrace theutilization of a computer device in designing educational lessons,implementing educational lessons, performing analysis and/or providingmodifications. An individual learner/student may interface with thecomputer device in the educational environment and may additionallyinterface with an instructor. If the instructor (or a peer) is at adistant location, their collaboration may be mediated through multiplecomputers connected to the interne or some other network.

Accordingly, FIG. 1 and the corresponding discussion are intended toprovide a general description of a suitable operating environment inwhich the invention may be implemented. One skilled in the art willappreciate that the invention may be practiced by one or more computingdevices and in a variety of system configurations, including in anetworked configuration.

Embodiments of the present invention embrace one or more computerreadable media, wherein each medium may be configured to include orincludes thereon data or computer executable instructions formanipulating data. The computer executable instructions include datastructures, objects, programs, routines, or other program modules thatmay be accessed by a processing system, such as one associated with ageneral-purpose computer capable of performing various differentfunctions or one associated with a special-purpose computer capable ofperforming a limited number of functions. Computer executableinstructions cause the processing system to perform a particularfunction or group of functions and are examples of program code meansfor implementing steps for methods disclosed herein. Furthermore, aparticular sequence of the executable instructions provides an exampleof corresponding acts that may be used to implement such steps. Examplesof computer readable media include random-access memory (“RAM”),read-only memory (“ROM”), programmable read-only memory (“PROM”),erasable programmable read-only memory (“EPROM”), electrically erasableprogrammable read-only memory (“EEPROM”), compact disk read-only memory(“CD-ROM”), or any other device or component that is capable ofproviding data or executable instructions that may be accessed by aprocessing system.

With reference to FIG. 1, a representative system for implementing theinvention includes computer device 10, which may be a general-purpose orspecial-purpose computer. For example, computer device 10 may be apersonal computer, a notebook computer, a personal digital assistant(“PDA”) or other hand-held device, a workstation, a minicomputer, amainframe, a supercomputer, a multi-processor system, a networkcomputer, a processor-based consumer electronic device, or the like.

Computer device 10 includes system bus 12, which may be configured toconnect various components thereof and enables data to be exchangedbetween two or more components. System bus 12 may include one of avariety of bus structures including a memory bus or memory controller, aperipheral bus, or a local bus that uses any of a variety of busarchitectures. Typical components connected by system bus 12 includeprocessing system 14 and memory 16. Other components may include one ormore mass storage device interfaces 18, input interfaces 20, outputinterfaces 22, and/or network interfaces 24, each of which will bediscussed below.

Processing system 14 includes one or more processors, such as a centralprocessor and optionally one or more other processors designed toperform a particular function or task. It is typically processing system14 that executes the instructions provided on computer readable media,such as on memory 16, a magnetic hard disk, a removable magnetic disk, amagnetic cassette, an optical disk, or from a communication connection,which may also be viewed as a computer readable medium.

Memory 16 includes one or more computer readable media that may beconfigured to include or includes thereon data or instructions formanipulating data, and may be accessed by processing system 14 throughsystem bus 12. Memory 16 may include, for example, ROM 28, used topermanently store information, and/or RAM 30, used to temporarily storeinformation. ROM 28 may include a basic input/output system (“BIOS”)having one or more routines that are used to establish communication,such as during start-up of computer device 10. RAM 30 may include one ormore program modules, such as one or more operating systems, applicationprograms, and/or program data.

One or more mass storage device interfaces 18 may be used to connect oneor more mass storage devices 26 to system bus 12. The mass storagedevices 26 may be incorporated into or may be peripheral to computerdevice 10 and allow computer device 10 to retain large amounts of data.Optionally, one or more of the mass storage devices 26 may be removablefrom computer device 10. Examples of mass storage devices include harddisk drives, magnetic disk drives, tape drives and optical disk drives.A mass storage device 26 may read from and/or write to a magnetic harddisk, a removable magnetic disk, a magnetic cassette, an optical disk,or another computer readable medium. Mass storage devices 26 and theircorresponding computer readable media provide nonvolatile storage ofdata and/or executable instructions that may include one or more programmodules such as an operating system, one or more application programs,other program modules, or program data. Such executable instructions areexamples of program code means for implementing steps for methodsdisclosed herein.

One or more input interfaces 20 may be employed to enable a user toenter data and/or instructions to computer device 10 through one or morecorresponding input devices 32. Examples of such input devices include akeyboard and alternate input devices, such as a mouse, trackball, lightpen, stylus, or other pointing device, a microphone, a joystick, a gamepad, a satellite dish, a scanner, a camcorder, a digital camera, and thelike. Similarly, examples of input interfaces 20 that may be used toconnect the input devices 32 to the system bus 12 include a serial port,a parallel port, a game port, a universal serial bus (“USB”), a firewire(IEEE 1394), or another interface.

One or more output interfaces 22 may be employed to connect one or morecorresponding output devices 34 to system bus 12. Examples of outputdevices include a monitor or display screen, a speaker, a printer, andthe like. A particular output device 34 may be integrated with orperipheral to computer device 10. Examples of output interfaces includea video adapter, an audio adapter, a parallel port, and the like.

One or more network interfaces 24 enable computer device 10 to exchangeinformation with one or more other local or remote computer devices,illustrated as computer devices 36, via a network 38 that may includehardwired and/or wireless links. Examples of network interfaces includea network adapter for connection to a local area network (“LAN”) or amodem, wireless link, or other adapter for connection to a wide areanetwork (“WAN”), such as the Internet. The network interface 24 may beincorporated with or peripheral to computer device 10. In a networkedsystem, accessible program modules or portions thereof may be stored ina remote memory storage device. Furthermore, in a networked systemcomputer device 10 may participate in a distributed computingenvironment, where functions or tasks are performed by a plurality ofnetworked computer devices.

While those skilled in the art will appreciate that the invention may bepracticed in a variety of computing environments, with many types ofcomputer system configurations, including networked environments, FIG. 2represents an embodiment of the present invention in a networkedenvironment that includes a variety of clients connected to a serversystem via a network. While FIG. 2 illustrates an embodiment thatincludes multiple clients connected to the network, alternativeembodiments include one client connected to a network, one serverconnected to a network, or a multitude of clients throughout the worldconnected to a network, where the network is a wide area network, suchas the Internet. Moreover, embodiments of the present invention embracenon-networked environments, such as where a dynamic continualimprovement educational, environment utilizes a single computer device.At least some embodiment of the present invention further embrace atleast a portion of a dynamic continual improvement environment that doesnot require a computer device.

In FIG. 2, a representative networked configuration is provided forwhich learning occurs. Server system 40 represents a systemconfiguration that includes one or more servers. Server system 40includes a network interface 42, one or more servers 44, and a storagedevice 46. A plurality of clients, illustrated as clients 50 and 60,communicate with server system 40 via network 70, which may include awireless network, a local area network, and/or a wide area network.Network interfaces 52 and 62 are communication mechanisms thatrespectfully allow clients 50 and 60 to communicate with server system40 via network 70. For example, network interfaces 52 and 62 may be aweb browser or other network interface. A browser allows for a uniformresource locator (“URL”) or an electronic link to be used to access aweb page sponsored by a server 44. Therefore, clients 50 and 60 mayindependently access or exchange information with server system 40.

As provided above, server system 40 includes network interface 42,servers 44, and storage device 46. Network interface 42 is acommunication mechanism that allows server system 40 to communicate withone or more clients via network 70. Servers 44 include one or moreservers for processing and/or preserving information. Storage device 46includes one or more storage devices for preserving information, such asa particular record of data. Storage device 46 may be internal orexternal to servers 44.

In the illustrated embodiment of FIG. 2, the networked system is used toprovide a dynamic continual improvement educational environment. Inparticular, the networked system provides dynamically gathers data,automatically adapts to an individual's pace of learning, selectivelydetermines the type and difficulty of content provided to an individual,selectively provides an exposure frequency for the content, and/orenables rapid design modifications within the educational environment,as will be further discussed below. Those skilled in the art willappreciate that the networked system of FIG. 2 is a representativesystem in accordance with the present invention. Accordingly,embodiments of the present invention embrace other computer systemconfigurations for performing methods disclosed herein.

Dynamic Continual Improvement Educational Environment

As provided above, embodiments of the present invention take place inassociation with a dynamic learning process that includes the ability todesign or develop an educational experience or otherwise provideeducational lessons (e.g., educational activities, educational content,etc.). The designing of the educational experience or lesson isfacilitated by utilizing an object oriented format, a drag-and-dropinterface, or another design process that eases development. Oncedesigned, implementation of the educational lesson is experienced by anindividual learner/student/tutor/administrator/user, and includesproviding instruction focused to the user and gathering data relating tothe user's learning. An analysis is performed on the data to optimizelearning and may correspond to a particular individual, group, and/oreducational lesson. Modifications are selectively or automatically madeto the educational lesson. The process of designing, implementing,analyzing, and selectively modifying creates a cycle that optimizes thelearning process and adapts to groups and/or individual learners withthe goal of improving learning outcomes and efficiency.

As provided herein, within the environment the educational lesson orcontent is dynamically provided to the learner on an iterative basisaccording to the need of that learner in the learning process. Forexample, a systematic spaced review of content is dynamically providedto the learner/student based on the student's accuracy and speed inunderstanding educational concepts/lessons taught. The systematic spacedreview facilitates retaining the educational concepts/lessons taught andcustomizes the teaching to the individual learner's ability to grasp theeducational concepts taught. The systematic spaced review of contentrelates to how frequent and over what duration of time a learner sees anitem based on accuracy and speed of the learner. Accordingly, it tracksthe accuracy and speed of understanding concepts, and transitions theconcepts from short-term to long-term memory.

Thus, learner performance data is gathered and is selectively used toadjust the pace of learning, to modify the frequency of exposure toparticular content, to control the sequencing of learning activities andto regulate the type and difficulty of content to which the learner isexposed.

Aspects of the educational environment are easily and/or automaticallyadapted to a learner's performance. For example, one type of adaptivityin accordance with embodiments of the present invention includesproviding an evaluation/pre-test prior to an educational lesson. Thelesson is then filtered down to include only the portions that the userfailed or did not know during the pre-test.

Another type of adaptivity in accordance with embodiments of the presentinvention relates to providing an evaluation prior to an educationallesson. Parallel versions of the same educational content are createdand the user takes a pre-test to determine which is most suitable forhis/her needs. After the version is selected, either automatically or byan instructor, the learner pursues only that version throughout theinstruction.

Another type of adaptivity in accordance with embodiments of the presentinvention relates to branching or cascading. In this type of adaptivity,the user is able to selectively navigate through the educational contentas the user desires. Evaluation is periodic and occurs at predeterminedpoints to determine when and what type and difficulty of content ispresented to the learner. At each evaluation point, there are a finitenumber of options that lead to potentially many more evaluation points,each with a finite number of options. In some embodiments, there aremany ending points to the various branches, but the number is ultimatelyfinite, unless an infinite loop is created (which is generally notdesired). The total number of original paths through the material ispredictable and finite (with the exception of the number of times a usermay go through an infinite loop).

Another type of adaptivity in accordance with embodiments of the presentinvention relates to computer adaptive testing. This includes continuousand ongoing evaluation. Test items are preset and ranked. Educationalcontent having a range of difficulty is presented to the student. Thestudent's responses to the educational content assist in establishing aparticular area of difficulty within the range for that student.

Another type of adaptivity in accordance with embodiments of the presentinvention relates to a performance feedback loop. This providescontinuous and ongoing evaluation. Speed and accuracy of every responseare evaluated. The instruction is not preset, and thus the learner cantake a near infinite number of paths through activities and educationalcontent based upon the learner's performance. This and any of the typesof adaptivity described above, can be used alone or in combination withany of the other type of adaptivity described above, as needed, and areall enabled by embodiments of the present invention.

With reference now to FIG. 3, a representative relationship within acontinual improvement educational process is illustrated. In FIG. 3,continual improvement educational process 80 includes a relationshipbetween development module 82, implementation module 84, implementationfidelity module 85, analysis module 86 and modification module 88. Inaccordance with the illustrated embodiment, development module 82includes a variety of tools (e.g., activity builder, adaptive pathbuilder, theme designer, audio cutter, audio start and end, conceptentry tool, concept type definer, component manager, content checker,course extractor, example tagger, gloss linker, media manager, objectmanager, relationship linker, rights management system, sentencesynchronizer, text importer, tool menu, translation editor, etc.) thatmay be employed to selectively design or otherwise provide instructionaldevelopment and/or content development. In at least some embodiments ofthe present invention, development module 82 is utilized to developeducational activities and/or content. Once designed, implementationmodule 84 is employed to implement educational content and datagathering. For example, in at least some of the embodiments of thepresent invention implementation module 84 is used to instructindividual students/learners and selectively gather data. Implementationmodule 84 includes a variety of tools (e.g., installer, instructionengine, learner guidance system, learning optimizer, collaborativeactivity manager, user/group manager, positive feedback generator,remote document feedback, rights management system, research organizer,tutor guidance system, updater, administrator guidance system, reportsbuilder, etc.) to enable implementation of educational lessons orcontent and/or to manipulate data.

Implementation module 84 interfaces with implementation fidelity module85 to determine the degree that implementation fidelity as defined indevelopment module 82 is achieved, and to provide administrators,teachers and tutors, and learners with information and instruction onhow to improve fidelity. Implementation module 84 also providesdesigners and researchers with data that can help them determine how toimplement the system and/or enhance the system.

Implementation module 84 also interfaces with analysis module 86, whichis employed to evaluate the learning. Analysis module 86 includesvarious tools (e.g., learning optimizer, research organizer, etc.) toperform the evaluation. Based on the evaluation, modification module 88may be employed to selectively customize educational content. Forexample, in one embodiment, response measurements are tracked from agiven experiment and are provided to research organizer for automateddata analysis, which can then be interpreted by an instructionaldesigner in order to improve the learning system. Or the results may befed to the learning optimizer for automatic adjustment of the learningsystem. In at least some of the embodiments of the present invention,one or more of the tools of development module may be utilized inmodification module 88.

The educational content may be customized in a variety of manners. Forexample, the frequency of which the content is presented may be modifiedto a learner or group, the order in which the content is presented maybe modified to a learner or group, or the difficulty of the content maybe modified to a learner or group. In some embodiments, modificationmodule 88 includes all the development tools listed under thedevelopment module 82.

Accordingly, embodiments of the present invention embrace theutilization of a variety of modules that enable the creation of adynamic and customizable continual improvement educational process. Adiscussion of each of the modules and corresponding tools will bediscussed independently below.

I. DEVELOPMENT TOOLS

In accordance with embodiments of the present invention, aninstructional designer selectively develops instructional/educationalcontent for a learner/student. As will be provided herein, thedevelopment of the educational content is facilitated by not requiringthat the educational content be modified at the code level by a computerprogrammer. Instead, a method (e.g., object oriented method,drag-and-drop method, etc.) is employed that enables an instructionaldesigner to quickly, dynamically and customizably create educationalcontent for a student/learner, or group of students. A variety of toolsare available for use by the instructional designer to develop thecontent. Accordingly, the following is a discussion relating torepresentative development tools available for use in designingeducational content in a continual improvement educational process.

A. Activity Builder

An activity builder tool is a development tool that lets instructionaldesigners and content experts selectively develop learning activities.With the use of an activity builder tool, the instructional designer cancreate individual test items, questionnaire items, menus, navigationbars, peer-practice and practice-with-tutor activities, a learnerguidance system, and/or tutor and administrator guidance systems. Anactivity builder tool generates activities for the learner at runtime.

With reference now to FIGS. 4-5, a representative user interface inassociation with an activity builder tool is illustrated. In FIG. 4, oneor more components are selected and dragged into a work area. In oneembodiment, the sequence of dragging the components relates thecomponents to each other. The components can also be related to one ormore variables. Further, an instructional designer is able toselectively modify properties of a particular component. In FIG. 5,collaborative activities are built.

In one embodiment, the activity builder tool includes a variety offeatures. For example, such features include specialized propertyeditors, read-only properties, abilities to handle complex components,support for multiple learning contexts, abilities to handle variableslike components, support for multi-component selection, a lock button tolock sizes and positions, support for experimental variables, andhandling of syntax errors in expressions. Further features includesupport for multiple layers of components, support for multiple layouts,various layout tools for alignment and sizing, direct componentmanipulation within the activity editing window, customizablehierarchically organized components list, shareable component folders,multi-user activities, an advanced mode that revealsless-frequently-used properties, zoom, ways of connecting components toeach other easily, a diagnostics feature, support for arithmetic andBoolean expressions in property values, rights management,human-readable display of property values, support for user datatracking, and component modules.

An activity builder allows an instructional designer trained in the toolto rapidly construct a limitless variety of sophisticated learningactivities without the need for time-consuming computer programming.There are many features of activity builder that make this possible,including connecting components without the use of a programminglanguage or scripting language, providing support for creatingcollaborative activities, utilizing a wide variety of sophisticatedcomponents, re-using functionality through component modules and dynamicactivities, providing features for flexible user data tracking andreporting, and supporting multiple layouts. These six features aredescribed below.

Existing authoring systems and rapid application development (RAD)environments typically require the tool user to use a programminglanguage or scripting language to achieve complex interactions betweencomponents. This necessitates a considerable time investment beforebecoming proficient with such a system, and mastery of such a system istypically only within the reach of a computer programming professional.Even after the initial time investment, the need for programming orscripting lengthens the development process. In contrast, activitybuilder intentionally prevents the use of scripting and programming, andinstead provides simple ways of creating complex interactions betweencomponents. One way is to drag and drop from the variables list to theproperties list. For example, to make a label in the activity visiblewhen a button is clicked by the learner, the instructional designersimply drags and drops the button's .clicked property from the variableslist onto the label's .visible property in its properties list. Then atruntime, when the learner clicks the button, its .clicked propertybecomes true, which in turn sets the label's .visible property to true,causing it to become visible to the learner. The variables list includesall of the activity's components and their properties, so theinstructional designer can easily set any property equal to any otherproperty by a drag-and-drop technique.

Alternatively, the instructional designer may connect components byselecting one component, and then clicking a second component whileholding the Alt key down. A connection dialog box then appears. Thisdialog box shows the most likely property connections between the twocomponents, based on known frequencies of component connections. Theinstructional designer may then click OK to simply accept the automaticconnections, or change the property connections via drop-down lists foreach property. For the sake of efficiency, in one embodiment thedrop-down lists just display the other component's properties having acompatible data type.

A third method of connecting properties is to use the drop-down listfound in the property value cells of the properties list. Thesedrop-down lists display every compatible property of all othercomponents and variables, including system constants such as true andfalse. After connecting two components, a blue arrow is automaticallydrawn between them in the activity editing panel. This blue arrowdepicts the flow of information from one component to the other, and isonly visible at design time. Properties can also be set to expressionsinvolving any number of properties. These expressions can includeBoolean operators (and, or, not, =, <, >, etc.), arithmetic operators(+, − /, *, etc.), parentheses, numbers, text, user properties,variables, and so on. This allows the instructional designer moreflexibility without introducing too much complexity. Continuing theprevious example, to make a label invisible when a button is clicked,the instructional designer would simply add “not” before the referenceto button.clicked. In another example, the instructional designer maywant to let the learner control the speed of an activity via a slidercomponent. In this activity, a delay timer component may be used to seta delay between two events. The delay timer's .startSeconds propertycould be set to the slider's .value property to control the speed.Alternatively, the delay timer's .startSeconds property could be set toslider.max-slider.value to reverse the orientation. Alternatively, theinstructional designer could set the .startSeconds property toslider.value*10 to increase the speed range.

The instructional designer can also create complex interactions betweenmultiple users, by creating collaborative activities in activitybuilder. To do this, the instructional designer specifies the number ofroles in the collaborative activity, and the number of participants foreach unique role. Activity builder then splits the activity editingpanel, so that the instructional designer can specify what each rolewill see. In essence, each role gets its own activity, with its owncomponents. What each participant sees and interacts with may be verydifferent during the collaborative activity, as specified by thedesigner. Components may be connected between the various roles, just asthey are connected within an activity, as described above. Thus, a tutorat a remote location may click a button that causes a graphic to appearon the learner's screen. Also two peers may collaborate on a gapactivity, where each learner lacks information that the other learnerhas. For example, one learner may have one half of a map, while anotherlearner sees the remaining half. The activity invites the two learnersto work together by exchanging information via text chat or audioconferencing, until they are both able to sketch the missing portion.Activity builder makes it feasible for an instructional designer tocreate such an activity rapidly. Each role is simply given a graphiccomponent with a different map image. The instructional designer mayalso give the learners the ability to see each other's sketches inprogress, by simply connecting a second graphic component to the otherlearner's sketch, via their image properties. At runtime, theinstruction engine can send this information in real time between theparticipants' computers, via the internet, LAN or other availablenetwork.

Another important difference between activity builder and existingdevelopment tools is that most other tools provide mostly generalpurpose “nuts and bolts” components. In contrast, activity builderspecializes in providing an extensive variety of sophisticatedcomponents tailored to designing adaptive instructional activities. Thisgreatly increases development efficiency. For example, the instructionaldesigner may use a multiple choice component to construct a multiplechoice exercise in a matter of seconds or minutes, because the multiplechoice component inherently provides option buttons for the learner toclick on, randomizes the options automatically, gives correctivefeedback, and so on. Another example is the dialog component, whichdisplays a dialog with accompanying audio. The dialog componentinherently provides audio buttons for the learner to click on in orderto listen to any sentence or paragraph. The dialog window also has a.clickedWord property which can be easily sent to a detailed translationcomponent to provide a translation or definition of any word the learnerclicks on.

Activity builder also increases development efficiency through componentmodules. Component modules are re-usable portions of an activity, havingcomponent(s) with pre-set property values and interconnections.Component modules are listed in the components list panel withcomponents, making them easily accessible to instructional designers todrag and drop into any activity. The instructional designer can createany number of component modules within activity builder, with the sameprocess used to create activities. By supporting component modules,activity builder allows the instructional designer to apply objectoriented principles such as encapsulation and abstraction. For example,by placing sub-components in a hidden layer, the inner complexity of thecomponent module can be hidden from the instructional designers thatsubsequently incorporate the component module in their activities. Theuse of component modules improves development efficiency, byencapsulating re-usable activity functionality and making it readilyaccessible. Component modules also facilitate efficient implementationof user-interface standards, since they can be pre-set to the prescribedsize, position, color, etc. A related aspect of activity builder is itssupport for dynamic activities. A dynamic activity is an activity thatdraws information from the properties of the concept the learner iscurrently learning. In other words, some of the properties of theactivity's components are set to values of the current concept'sproperties, which are resolved at runtime. For example, a dynamicvocabulary activity may contain a label with the word being learned andanother label for its translation. In the present example, the firstlabel's .text property would be set to currentConcept.name, and thesecond label's .text property would be set tocurrentConcept.translation. This single activity could then be used inlearning any number of vocabulary concepts, since the contents of theactivity are dynamic. Instructional design efficiency is significantlyincreased by activity builder's support for dynamic activities. Also,since concept type definer (described below) allows the instructionaldesigner to create any number and variety of concept properties, theadvantage of dynamic activities is greatly enhanced.

Activity builder also provides a flexible method for adding user datatracking and reporting. Specifically, a time measurer component allowsthe instructional designer to track the duration of activityinteractions with the learner. A counter component provides tracking ofthe number of times a given interaction or group of interactionsoccurred. For reporting, activity builder provides a variety of chartingcomponents for displaying bar charts, pie charts, line graphs, tables,and so on. Also, to supply data to these charts, activity builder offerscomponents that query for and process user data previously collected.The instructional designer can use any of these components incombination to create activities that supply reports to learners,tutors, and administrators.

In addition, activity builder simplifies the creation of multipledelivery formats. To accomplish this, activity builder allows theinstructional designer to specify any number of target layouts. Theselayouts might include a desktop PC, PDA, cell phone, or 8½×11 printout.The components in a given activity can then be sized and positioned foreach layout separately, while all other aspects of the activity remainthe same. This boosts development speed considerably in cases where theinstruction is targeted at multiple delivery platforms. Also, activitybuilder provides tools to speed up the sizing and positioning for eachlayout. Specifically, in the moment the instructional designer switchesto a new layout of an activity for the first time, activity builderautomatically resizes and repositions the components in the activity tofit the new layout. In one embodiment, the instructional designer canselect from one of four options: (1) auto-size and position everycomponent proportionally so that the new percentage sizes and positionsrelative to the whole activity are the same as the previous percentagesizes and positions, (2) the same as above, except that components with.keepAspectRatio set to true would maintain their aspect ratio byshrinking and centering themselves to fit within the region that thefirst algorithm would have allowed, (3) the same as above, but keep theaspect ratio intact for all components, or (4) maintain the aspect ratioand relative positions of all components, and simply center andshrink/expand the entire activity to the maximum size permitted by thenew layout. The instructional designer may then opt to refine thecomponent sizes and position for each layout as needed.

As indicated, the activity builder allows instructional developers tocreate learning activities. It includes a drag and drop interface forassembling activities from components and facilitates dynamic, adaptiveinstruction. It provides a list of dynamic content elements, which canbe connected to components. It further provides a way to adjustalgorithms for different learners. These algorithms can affect thebehavior of various components and content elements.

Activity builder may also be used to create menus, navigationbackgrounds, peer-practice and practice-with-tutor activities, a learnerguidance system, tutor and administrator guidance systems, questionnaireitems, test items, and reports. The activity builder includes codeassociated with drag and drop objects to generate activities for the enduser. A portion of this same code is used at runtime to deliver theactivity to the learner.

There are three types of users of the activity builder: (i)instructional designers, (ii) graphic designers, and (iii) contentexperts. The instructional designer uses the activity builder todetermine what elements will appear on the screen, what responses areinvited from the learner, and how the responses are judged. The graphicdesigner decides the layout of the elements, and the graphics and soundsassociated with each component. For activities that allow staticcontent, the content expert or instructional designer uses the activitybuilder to enter this content.

The activity builder is generally opened from the adaptive path builder,which will be discussed below, though it can also open directly via thetool menu. Activities can be edited, previewed, copied or created as theneed arises.

As illustrated in FIGS. 4-5, representative features associated with theactivity builder include a button bar, an instructional design mode, alayout mode, a content mode, a test mode, components, content elements,properties, variables, ratings, an expression evaluator, notes, acomponent magnifier, runtime issues, an XML parser, and errorprevention. In an instructional design mode, the activity builder allowsa designer to choose which components should appear within the activity.In one embodiment, activities scale themselves automatically to anyheight and width, with a fixed aspect ratio specified in the layout. Theaspect ratio is always maintained, and the components in the activityare always scaled proportionally. Content elements gather contentdynamically from the content database, and then supply this content to acomponent of the activity for display.

B. Adaptive Path Builder

The adaptive path builder tool allows instructional designers to developthe sequence of activities for learning a task or concept. It alsoallows designers to organize test items to assemble a test, set thetiming of questionnaire items, determine which concepts precede otherconcepts, such as via a drag and drop interface.

With reference now to FIG. 6, a representative user interface inassociation with an adaptive path builder tool is illustrated. In FIG.6, the user interface includes selectable content, variables, concepts,properties, and an adaptive path editing window. The adaptive pathediting window is where the current adaptive path can be seen. Itdepicts the flow of activities used to teach a certain concept. Theconcept is what is being learned, while the adaptive path is how it islearned. Also, an adaptive path depicts which concepts should be learnedbefore other concepts.

Accordingly, an adaptive path builder allows an instructional designerto specify how a given concept should be learned and/or tested. Anadaptive path can be a linear sequence of activities or a more adaptivesequence with individualized branching and repetition, such as acomputer-adaptive test. To achieve systematic spaced review, activitiesare separated by delays (e.g., minutes, hours, days, etc.). Also, anadaptive path indicates which concepts should be learned or testedbefore others. Branching and properties are based on the individual'slearning context, which includes any known characteristics of thelearner. For automated experiments, the branching conditions and/orlearning context may include experimental variables defined by thedesigner or researcher.

Tests and questionnaires are created as adaptive paths. To get learnerfeedback on a particular activity, a questionnaire activity is placedafter the activity within the adaptive path. By indicating branchingconditions, the questionnaire activity is activated a certain percent ofthe time for a group of learners, or only when a learner has struggledwith a previous activity.

An instruction engine (in concert with a learner guidance system) usesthe information in the adaptive paths to recommend what the learnershould learn next. The learner may skip any activity in any adaptivepath. When an activity is skipped, the learner guidance systemautomatically asks the learner why he/she skipped the activity. Thelearner's reasons may include that it was “too easy,” “too hard,” “tooboring,” “too confusing,” “to repetitive,” “I already mastered thisconcept” or “the activity is not working right.” This feedback appearsas ratings for the activity and adaptive path, and can lead toimprovements made in the instruction.

Property values in an adaptive path may be part of an experiment createdin research organizer. The learning optimizer optimizes and setsexperimental variables that are defined in an adaptive path based onanalyses of previous experiments.

For efficiency, a relationship between concepts of the educationalcontent may be first defined in a relationship linker tool. (Therelationship between the concepts can be referred to as a concept link.)A relationship linker (discussed below) allows the instructionaldesigner to link or otherwise relate concepts together so that they arelisted together in the concept list of the adaptive path builder, anddisplayed together on the adaptive path. Concepts can be related bothhierarchically and in predecessor-successor pairs, with any number oflevels desired in the hierarchy, both upward and downward, and with anynumber of concepts on either side of a relationship.

For purposes of discussion herein, the hierarchy includes one or moreconcepts that are presented in the adaptive path before one or moresubsequent concepts of that same adaptive path. Accordingly, the priorconcepts can be referred to as “predecessor concepts” and the subsequentconcepts can be referred to as “successor concepts” to establish therelationship of the concepts in the adaptive path.

The reports builder, which will be discussed below, draws upon data thatis tracked automatically as the learner uses adaptive paths. Forexample, this includes data such as: time spent in the adaptive path,time spent completing an activity of each adaptive path, time spent in astage of each adaptive path, idle time in all the above areas, andwhether a given activity is completed or skipped.

An adaptive path is tied to a single concept or concept type.Accordingly, two concepts do not share the same adaptive path, unlessthe designer copies and pastes it from one concept to the other. Aconcept type can have an adaptive path that all of its concepts use orpoint to by default.

Learner guidance system concepts, which determine the menus the learnerwill see at login and will be discussed below, can have an adaptivepath. Different learners may get different learner guidance systemactivities, such as a progress list and a to-do list. This is determinedby the branching conditions within the adaptive path. The learnerguidance system also allows individuals to schedule activities with eachother, which implies that there are global adaptive paths that can beused to conduct these spontaneously scheduled events.

As illustrated in FIG. 6, features of the adaptive path builder includea button bar, an adaptive path editing window, a concept list, anactivity list, a branching window, a properties window, a variableswindow, ratings, notes, an activity preview and an error prevention. Inone embodiment, all of the tools are multilingual. Accordingly, when anindividual runs the instruction builder, it prompts the individual tologin. Each user has a primary native language, and this native languageis used by default throughout the tools.

In the illustrated embodiment, the adaptive path may be edited for adifferent concept by clicking on any concept or concept type in theconcept list and then clicking on the edit button in the button bar. Ifa concept has more than one adaptive path, the designer will be promptedto choose one to edit.

In FIG. 6, activities are arranged left to right, from beginning to moreadvanced. Adaptive paths begin and end with stage markers, such as“Introduction” and “Mastery”. The stage markers can be given any name,and any number of stage markers may be created.

The stage markers break the activities up into meaningful stages oflearning. The first stage marker branches to any number of activitieswithin the adaptive path, which means that the first activity actuallyencountered by the learner could be anywhere within the adaptive path.In the illustrated embodiment, the adaptive path can be scaled to anysize, so that the entire path can be viewed all at once if desired.Further, in at least some embodiments of the present invention, thedesigner can copy and paste entire sections of activities and branchingfrom one adaptive path to another. This is performed by selecting anysection of an adaptive path (e.g., by clicking and dragging a selectionbox around the area you want to select). A placeholder activity box oradaptive path box may be used that can be defined later.

A designer can indicate that an activity or concept link is required inthe branching window. In one embodiment, optional branches are depictedwith dashed arrows, whereas required branches are depicted with solidarrows. When branches are added, they are set to ‘required’ by default.To some extent, the exact placement and appearance of icons in theadaptive path layout is automatic. For example, icons always snap to agrid. Within certain constraints, a designer can move, insert, anddelete activities and other icons within the adaptive path.

In FIG. 6, when an icon is moved, all branches in and out of the iconare maintained since the arrows move to the icon's new position. Adesigner can indicate which activities branch to other activities in theadaptive path. A branching arrow is automatically drawn between twoactivities and a new text entry line appears in the branching window.The curvature, path, and line style of arrows is automatic. In FIG. 6,the diamond shapes represent branching decision points. A designer canclick and drag from a diamond to another activity to make a connection,or can just click on the diamond and then on an icon to make aconnection. If the designer clicks on a diamond and then changes his/hermind, the designer can click the background so that no branch iscreated.

Multiple parallel activities can be scheduled for a learner after anactivity is performed. In the illustrated embodiment, when a designerwants several concept links or activities to occur after a certainconcept link or activity, the designer uses a circle instead of adiamond to designate the relationship. Many branches can be taken from acircle, whereas only a single branch can be taken from a diamond.

A trailing arrow appears while adding a branch. When a concept link isadded to an adaptive path, the thumbnail of its adaptive path isinserted in the adaptive path that is being edited. When the designeradds a concept link to an adaptive path, a relationship is automaticallyformed between the two concepts if it does not already exist. When anadaptive path is opened, a check is made for concepts that have a“part/whole” relationship with the current concept. These relationshipsmay have been created by the designer in a relationship linker, whichwill be discussed below. Concept links for child concepts are addedautomatically to the adaptive path, if they do not already exist.

In establishing relationships between concepts, a branch may be formedautomatically or added manually. The relationship can indicate that aparticular concept link is optional. Any infinite loops that existbetween concept links are reported to the designer.

As new content layers are created in the activity builder, they appearby default. (A content layer is any activity whose concept property isset to a specific concept.) A designer can double-click an activity toedit it using the activity builder.

If a designer adds one or more activities to an adaptive path, whereindynamic content elements are required in the adaptive path that theconcept doesn't support, then the designer is notified. For example, ared slash appears over the activity to inform the designer that theactivity is not functional. Also, activities that are missing contentare highlighted with a red slash over the activity thumbnail. If anactivity is non-functional for any reason, the red slash appears.

A designer can add activities with content layers from concepts otherthan the current concept. For example, this may be done by dragging theactivity from the activity list, and then changing the content layerproperty of the new activity icon. Further, a designer may add to theadaptive path any activities that are missing content layers, and thenadd the content layers later. Thus, just because an activity requires acontent layer to function, this doesn't mean a content layer must beincluded before the adaptive path can be saved.

Activities or concept links can be grouped into a set. This feature isparticularly useful when the designer wants the learner to randomly getone of several activities at a certain point in the adaptive path.Activity sets have their own special properties that govern whichactivities will actually be given to the learner. The set of activitiesis enclosed in a box. Activity sets have two properties associated withthem: “AllRequired,” which indicates that all of the activities in theset must be completed before the learner is allowed to proceed, and“NumberRequired,” which indicates that a designated number must becompleted before the learner can proceed.

To rotate systematically through a pool of questionnaire items, adesigner can instruct the set to give the first n questions the firsttime through the adaptive path, the second n questions the second timethrough, and so on. Accordingly, a bookmark may be kept that keeps trackof which activity the learner received most recently from the set.

For efficiency, a designer can define activities that apply tosub-concepts. For example, right beneath an activity thumbnail, thedesigner can indicate to which sub-concepts the activity applies. Adesigner can name a single sub-concept, or indicate that “allsub-concepts” should get the activity. In one embodiment, when the “allsub-concepts” option is selected, the activities are mixed closetogether in time so that the learner doesn't necessarily know whichsub-concept is currently being utilized. In at least some embodiments,activities of sub-concepts can be entirely defined in the adaptive pathof a parent concept. Additionally, the sub-concepts can have their ownadaptive paths.

In one embodiment, the designer indicates that a given single-personactivity should be done by the tutor, supervisor or proctor instead ofthe learner. In this way, a tutor-initiated task practice can beassigned to a tutor automatically when a learner reaches a certain pointin his or her adaptive path. Also, a “proctor unlock” activity can becreated and placed at the start of an adaptive path. This activity willprompt the proctor to unlock the test (i.e. unlock an adaptive pathwhich contains the test) for the learner.

An activity list is a list of all activities defined in activitybuilder, described above. A designer can drag and drop any activity intothe adaptive path editor at any point within the adaptive path.

In a branching window, a designer can set how branching will occur froma given activity. (This applies to any icon in the adaptive path, suchas stage markers, sets, concept links, etc.) The most recently selectedicon has the corresponding branches displayed from this icon. A branchis defined by seven parameters, namely nib type (diamond or circle),whether it is required, a condition, an X destination (horizontal jump),a Y destination (vertical jump), a minimum delay, and a maximum delay.

The nib type parameter is used to control how various branchesproceeding from a given icon relate to each other. In one embodiment, ablack circle nib is selected to indicate that each and every branch isto be followed in parallel, provided each condition is true, whereas ablue diamond nib allows only the first branch to be taken, whosecondition evaluates to true. Green and magenta diamond nibs behave likeblue diamond nibs. Multiple diamond nibs are used in cases where thereare multiple branching decisions to be made, resulting in takingmultiple branches in parallel.

A particular branch may be set to a ‘required’ status if thecorresponding learning activities are considered essential to learning aparticular concept. Otherwise, the branch may be set to optional status,in which case the branch is represented graphically with a dotted lineinstead of a solid line in one embodiment. During the learning process,the instruction engine will give the learner the option to take theseoptional branches, provided the branching conditions evaluate to true.

The branching conditions from an icon are listed together in thebranching window, each on a separate line, one for each possible branch.Branching conditions are Boolean expressions that may include anycombination of variables and Boolean operators. In this way, a designercan establish branches based on how well the learner completed theprevious activity, or any number of other characteristics of thelearner.

The X destination parameter controls the horizontal distance of thebranch. Generally, the adaptive paths are oriented to progress from leftto right. Thus, the further the branch jumps to the right, the morelearning progress the learner appears to be making. Forcomputer-adaptive testing and related applications, the X destinationmay be set to an expression having performance variables and arithmeticoperators, to make the branching even more dynamic and concise.

The Y destination parameter controls the vertical jump of the branch.Generally, the vertical orientation of the adaptive path is arbitrary.What is important is that the X and Y destinations are set such that agiven branch leads from one icon to another, resulting in an appropriateinstructional flow based on the learner's performance, background, andinterests.

The minimum delay parameter controls the amount of time the learner mustwait, before proceeding to the destination of the branch, from the timethe decision was made to take the branch.

The maximum delay parameter controls the amount of time the learner mayelect to wait, before taking the branch becomes urgent instructionally.For example, in the case of a vocabulary adaptive path that includessystematic spaced review, a minimum delay may be set to 12 hours, with amaximum delay set to 20 hours. The minimum delay ensures that thelearner does not cram the learning material. The maximum delay ensuresthat the learner reviews the vocabulary concept before forgetting itcompletely.

In at least some embodiments, backward branching is allowed. This is auseful way to ensure long-term retention. Thus, even after reaching amastery stage marker, a backward branching condition may provideperiodic review for that concept at increasing intervals. Specifically,the designer may include a variable (e.g., “PredictedDaysofRetention”)in the expression that establishes a delay. This applies to a givenconcept and may be based on the longest period between reviews where thelearner has been able to remember a particular concept.

In some embodiments, global navigation buttons allow a learner tooverride branching. Accordingly, the learner can selectively skip to thenext activity, go back to a previous activity, jump to a conceptexplanation activity, return to the main menu, exit the program, or thelike.

In some embodiments, a single branching arrow represents multiplebranching conditions that lead to the same activity. A number isdisplayed with the arrow to identify the number of branches represented.

A properties window of the adaptive path builder is analogous to aproperties window in activity builder, discussed above. Like thebranching window, the properties window applies to whichever activityhas the focus. An activity magnifier shows an enlarged thumbnail of theactivity that has the focus to allow the designer to more easily readthe contents of a given activity.

C. Theme Designer

The theme designer makes the system aesthetically pleasing and appealingto learners, and provides an environment in which media artists canquickly customize the look and feel of the environment to a particularaudience. A theme includes the underlying interface metaphor for thebuttons, text boxes, scrollbars, etc., a color scheme, a font scheme, agraphic scheme, and a sound scheme. In at least some embodiments, eachtheme in the system has the same number of elements as each other theme,even though the visual and audio presentation of the elements betweenthemes may be extremely diverse. For example, a designer could create auniversity theme, with formal, academic graphics, sounds, and colors oruse the same functionality to create a parallel beach theme, withinformal graphics, sounds, and colors. Because these items aredynamically presented via a database at runtime and there is aone-to-one relationship between the elements in the two themes, thetheme can be instantly changed on demand.

With reference now to FIGS. 7-8, a representative user interface isillustrated that corresponds to a theme designer tool in accordance withthe present invention. As illustrated, the theme may correspond toschool colors or another particular scheme. The theme designer includesa variety of components. For example, an author tool bar includes a toolselector dropdown to navigate to the theme designer, a new button, anopen button, a save as button, and a print button. A look selector isutilized by a graphic artist to develop the look and feel. A colorscheme selector/editor allows a graphic artist to select an existingcolor scheme or create a new color scheme. A font scheme selector allowsa graphic artist to choose an existing font scheme. A font scheme editorallows a graphic artist to create or modify a font scheme. A graphicscheme selector allows a graphic artist to select an existing graphicscheme. A graphic scheme editor allows a graphic artist to upload a newgraphic scheme. A sound scheme selector allows an audio artist to selectan existing sound scheme. A sound scheme editor allows an audio artistto upload a new sound scheme. A preview tab is used to apply a scheme toan activity. A style guide tab displays a sample of each graphic, color,look, font, and sound in a particular theme. A requirements tab is wherethe naming conventions for graphics and sounds are specified along withany other policies regarding themes.

A variety of tools interact with the theme designer or otherwiseinfluence the look and feel of the environment. For example, theadaptive path builder allows instructional designers to select thethemes used in the path according to learning context. The activitybuilder allows graphic designers to define multiple layouts for a givenactivity and allows a designer to specify a background to be used with agiven layout.

D. Audio Cutter

An audio cutter tool cuts a long audio file into smaller files byautomatically detecting the gaps of silence between audio segments. Theuser can adjust the silence sound threshold and then automatically applyit to the entire file. Manual cutting and adjusting of the audio file isalso possible. The audio cutter tool further names a new file, saves thename of the file under the correct concept property value, saves thefile in the correct file folder, automatically inputs metadata, and thelike. With reference to FIG. 9, a representative user interface isillustrated that is associated with an audio cutter tool. This allows alarge quantity of concepts' audio to be cut up and processed in afraction of the time in would normally take using traditionaltechniques.

E. Audio Start and End

An audio start and end tool allows instructional designers or contentdevelopers to modify the start and end positions of a selected audiofile for static content purposes.

F. Concept Type Definer

A concept type definer tool organizes the main areas (i.e., concepttypes) of a course allows designers to associate concept types withrelationship types and properties that become available for definingconcepts of that type. In addition, properties can be assigned directlyto concept types that allow special functionality to assist in rapiddevelopment. (e.g. automatic linking in adaptive path builder forcertain concept types). As provided herein, “concepts” are what istaught and “concept types” are categories of concepts. (e.g., concepttype is “vocabulary” and the concept is “family”). This applies to everygranularity of content. Concept types include everything from domains oflearning and complete courses, to words and phrases. (e.g., concept typeis “course” and concept is “Basic English”). Concepts have attributesthat are called properties (e.g., audio file, translation, etc.).Concepts also have relationships that define how they are the same,different, or part of another concept (e.g., synonym, antonym, etc.)With reference now to FIG. 10, a representative user interface isillustrated that corresponds with a concept type definer tool. Asillustrated, the concept type, available relationship and availableproperties are provided. The concept types may be selectively used todevelop educational content.

G. Concept Entry Tool

A concept entry tool allows instructional designers or content expertsto input and link concepts to a concept type. It further allows users toattach properties values (e.g., audio files, graphic files, etc.). Tospeed up the filling in of property values, tools are made toautomatically facilitate the process. (e.g., audio cutter, translationeditor, etc.) To further assist rapid design property values of conceptsare used in activity builder to build dynamic activities. With referencenow to FIG. 11, a representative user interface is illustrated thatcorresponds to a concept entry tool. As illustrated, a listing ofconcepts is provided and may selectively be used in designingeducational content.

H. Component Manager

A representative component manager tool provides control or managementover components of the system.

I. Content Checker

A content checker tool identifies potential problems corresponding tocontent (e.g., missing content in an adaptive path or activity, unusedcontent, temporary content, etc.) and gives the user instructions on howto solve the problems. The content checker tool also identifies contentdevelopment priorities. With reference now to FIGS. 12-13, arepresentative user interface that corresponds with a content checkertool is illustrated.

J. Course Extractor

A representative course extractor tool selectively exports content torun a resource. In at least one embodiment, rather than all of thecontent being exported, the only content extracted is the relevantcontent. Accordingly, the system speed is enhanced.

K. Example Tagger

An example tagger selectively tags sentences and words as examples ofconcepts to be used dynamically in activities. The example taggerutilizes property values, relationships, and text in the searchcriteria. With reference now to FIG. 14, a representative user interfaceis illustrated that corresponds to an example tagger. In the illustratedembodiment, “work” is tagged and thus is provided in various conceptsand is identified in a variety of sentences.

L. Gloss Linker

A gloss linker tool enables a designer to assign a gloss to a word in asentence through a view of a given dialog or individual sentences. Thedesigner decides whether the meaning or function of a word is sodifferent that they want it separated into a new gloss (e.g., match canbe a noun or a verb). If the designer wants to make a distinction, thegloss linker is used to assign the correct gloss to a word in a givensentence. With reference to FIG. 15, a representative interface thatrelates to gloss linker is shown. When a dialog or sentence is importedthe gloss is automatically assigned where possible. The gloss linkerprovides a place for the designer to control the gloss assignment andmanually assign remaining un-glossed words. The different coloredhighlighting helps the designer know what work needs to be done on agiven dialog or sentence. The resulting gloss information can then bedynamically used in activities to help the learner know the exactmeaning of a word in any dialog or sentence. This information can alsodynamically be used to provide example sentences for a variety ofvocabulary building activities. Gloss functions to highlight the wordhighlights in the sentence. The use of the gloss can provide emphasis onthe word and/or can identify to the user a use of the word in thesentence.

M Media Manager

A media manager tool enables a designer to search for and select mediaobjects or text for use in the activity builder and facilitates thereuse of content. With reference to FIG. 16, a representative userinterface for association with a media manager tool is illustrated. Adesigner/user may utilize a media manager too to selectively search textby concept, example sentence, scenario, dialog, etc., or similarlysearch media objects by image, audio, video, or the like. Accordinglydesigners/users are able to selectively use media or other stored filesfrom a library. In FIG. 16, a media manager tool is utilized to obtain agraphical representation of a flag of the United States of America thatmay be used in educational content.

N. Object Manager

A representative object manager is user to manage groups, users, userroles, locations and/or stage markers within the system. By way ofexample, an object manager tool enables the following tasks to beperformed: (i) defining the properties of an object, including creating,editing and deleting object properties; (ii) creating new objects; and(iii) editing existing objects.

O. Relationship Linker

A relationship linker tool identifies a relationship shared by twoeducational concepts. With reference to FIG. 17, a representative userinterface that relates to a relationship linker is illustrated. In FIG.17, the relationships are selectively identified. In one embodiment,relationships that may be identified include “same as”, “differentthan”, “part of” or another relationship. The relationships available tothe designer for a given concept type are defined in a concept typedefiner, thus allowing the designer the capability of adding anyrelationship needed for a given course. These relationships can then beused dynamically in activities to speed up development. Somerelationships can have special functionality that is the same for anygiven course that help to display menus, and the like.

P. Rights Manager/Rights Management System

In accordance with embodiments of the present invention, a rightsmanager or rights management system may be associated with developmentmodule 82 and/or implementation module 84. A representative rightsmanager is used to control user access. For example, access may becontrolled for such objects as activities, concepts, adaptive paths,media, themes, concept types, etc. The control may be exercised in avariety of ways. Rights manager may be used to set each user'spermission to read, write and/or use system objects as well ascontrolling ownership of objects. Each user's permission to use systemtools in order to perform tool operations, such as using activitybuilder to build activities, may be controlled by rights manager.

Q. Sentence Synchronizer

A sentence synchronizer tool provides an efficient way to attach anaudio file to a dialog and identify audio starting and ending points ofthe dialog, paragraphs, and sentences. A representative user interfaceassociated with the sentence synchronizer is illustrated in FIG. 18. Thetool allows the user to select a dialog to work on, allows the user toattach an audio-video file to the dialog, automatically assigns startand end points for the dialog, allows the user to manually identify theend of every sentence, allows the user to manually modify the start andend points, and/or allows the user to link a graphic with any sentencein the dialog.

R. Text Importer

A text importer tool imports content intended for dynamic use. FIGS.19-20 illustrate a representative user interface giving thedesigner/user choice over types of content to import, characteristics ofcontent (i.e., needs translation and/or audio), and what to filter orignore while importing (i.e., case-sensitivity, punctuation, etc.) Inaccordance with the illustrated embodiment, the text importer tool loadsdialogs, sentences, and words into appropriate database tables,establishes links between words and their respective sentences,identifies and adds new words introduced by imported sentences, suggestswords that should be combined and/or allows the user to combine wordsthat have been combined in the past.

S. Tools Menu

A representative tools menu allows a user (e.g., an instructionaldesigner) to selectively obtain or launch the one or more tools neededto perform a task.

T Translation Editor

A translation editor tool provides translators and content editors withtext that is ready for translation and review. The text is identifiedand categorized automatically when text is imported in text importer,saved in activity builder, and/or input in concept entry tool, etc. Thetranslator is presented with the necessary information and context topermit high quality translation (e.g., shows activities, media assets,larger contexts, and associated concept properties). A representativeuser interface associated with a translation editor is illustrated inFIGS. 21-23.

U. Other Development Tools

Those skilled in the art will appreciate that the development toolsprovided herein are representative of tools available for use by adesigner to develop educational content for a dynamic continualimprovement educational environment. Accordingly, embodiments of thepresent invention embrace other development tools. An example of anothertool is illustrated in FIG. 24, which illustrates a representative userinterface that is associated with a component tester tool, whichexecutes automated tests on components to ensure that they functionproperly and selectively diagnoses or otherwise notifies a user of anerror or problem. Those skilled in the art will appreciate thatembodiments of the present invention embrace other tools that enable adesigner to develop educational content for a dynamic continualimprovement educational environment.

Thus, in accordance with embodiments of the present invention, aninstructional designer selectively develops instructional/educationalcontent for a learner/student. As provided herein, the development ofthe educational content is dynamic and efficient and is facilitated bynot requiring that the educational content be modified at the code levelby a computer programmer. Instead, a method (e.g., object orientedmethod, drag-and-drop method, etc.) is employed that enables aninstructional designer to quickly, dynamically and customizably createeducational content for a student/learner/tutor/administrator, or groupof students. As discussed above, a variety of tools are available foruse by the instructional designer to develop the content. Examples ofsuch tools include an activity builder, an adaptive path builder, atheme designer, an audio cutter, an audio start and end, a concept entrytool, a concept type definer, a content checker, an example tagger, agloss linker, a media manager, a relationship linker, a sentencesynchronizer, a text importer, a translation editor, and/or other suchtools that enable a designer to quickly, dynamically and customizablycreate educational content for one or more students/learners in adynamic continual improvement educational environment.

II. IMPLEMENTATION TOOLS

Once the instructional/educational content is developed, the content isselectively implemented in the dynamic continual improvement educationalenvironment. In at least some embodiments of the present invention,implementation of the educational content includes customizably anddynamically instructing individual students/learners and selectivelygathering data. A variety of tools (e.g., instruction engine, learnerguidance system, learning optimizer, collaborative activity manager,user/group manager, positive feedback generator, remote documentfeedback, rights management system, research organizer, reports builder,etc.) are available for use. Accordingly, the following is a discussionrelating to representative implementation tools available for use inproviding educational content in a dynamic continual improvementeducational process.

A. Instruction Engine

An instructional engine identifies which activities/lessons arecurrently available to the learner, and recommends or decides whichactivities/lessons to present next. With reference now to FIG. 25, arepresentative user interface associated with an instruction engine isprovided. In some embodiments, the learner guidance system provides thevisual interface instead, and the instruction engine works in thebackground to provide the learner guidance system information it dependson.

The instruction engine keeps track of the learner's progress in learningconcepts and initiates the instructional activities that are needed tofurther the learning progress. In some embodiments, when an activitybecomes available to the learner according to the branching of anadaptive path, the activity is added to the list of all availableactivities for that learner. In some embodiments, the role property ofthe activity may be set to someone other than the learner. Accordingly,activities or the results thereof are made available to anotherindividual or system, such as a tutor, instructor, manager, proctor,etc.

In some embodiments, an instruction engine supports multi-useractivities. Accordingly, interactive activities between multiple usersor a user and an instructor or tutor may be provided in the dynamiccontinual improvement educational environment.

In some embodiments, an instruction engine is used for purposes otherthan instruction. For example, there is a decision tree that a developermust use to choose the part of speech property of English word concepts.This decision tree can be defined as an adaptive path—a series ofquestions and branches. The instruction engine can thus execute theadaptive path, even though it's not related to the learning of aparticular concept.

In some embodiments, tutors utilize the instruction engine for otherpurposes. For example, when conducting a task practice the tutor will beguided through several activities that comprise the adaptive path fortask practices.

An instruction engine enables all of the features defined by an adaptivepath. Examples include branching conditions, delay ranges, etc. Aninstruction engine also invokes activities defined by Activity Builder.A learner guidance system works in harmony with an instruction engine byproviding the learner with prioritized options for what to learn next.Once the learner chooses an option, the instruction engine starts theactivities related to that option, one at a time until the learnerfinishes all activities related to the chosen option, until there aremore activities but they are all delayed, or until the learnervoluntarily returns to the main menu (learner guidance system) or exitsthe program.

In at least some embodiments, an instruction engine does not have itsown user interface. Instead, it invokes menu activities which can invokeother activities. The instruction engine maintains a list of availableactivities, chooses which activity to do next, and/or keeps track ofprogress. In addition, the instruction engine attends to issues relatedto learner control, multi-user activities, mid-stream changes toinstructional content, positive feedback, optimization, and/or providinglists of related concepts for multi-concept activities.

Once a learner registers for a course, the first stage marker on thecourse's adaptive path automatically becomes available to the learner.The first time the learner logs in after registering for a new course,the Instruction Engine evaluates the conditions of the branchesemanating from this stage marker. In this way, any number of activitiesor concept links can be made available to the learner from day one.

In evaluating branching conditions, as soon as any activity is completedor skipped, the instruction engine evaluates the branching conditionsfor the branches emanating from the activity. Likewise, as soon as anystage marker is reached, the instruction engine evaluates the branchingconditions for the branches emanating from the stage marker. However, inat least some embodiments, branching conditions of branches emanatingfrom concept links are evaluated only after the learner reaches thespecific stage of the concept from which the branches emanate. In oneembodiment, the branching conditions are evaluated immediately once aconcept link is encountered. In another embodiment, an instructionengine tracks the trail of the learner so that it will only evaluate thebranches for the concept link from which the learner originated.

In some embodiments, an instruction engine allows multiple concept linksto the same concept within an adaptive path and keeps track of whichconcept link led to a given activity. That way, when an exit branch isencountered for a concept link upon the learner reaching an underlyingstage marker, the instruction engine knows which concept link branchingconditions to evaluate. Furthermore, in at least one embodiment and as asimplifying constraint on the system, it is presumed that a given exitpoint will only be used once within an adaptive path. That way, if thereare two concept links to the same concept within an adaptive path, onlyone of the two concept links have a branch exiting its second stagemarker, and only one has a branch exiting its third stage marker, and soon.

In some embodiments, the branching conditions of branches emanating fromsets (e.g., activity sets, concept link sets, etc.) are evaluated afterthe learner has fulfilled the completion criteria defined by the set.

Branching can take place based on how the learner answered the previousquestion. For example, after a multiple choice question the branchingmight in essence be: “If option=A then . . . elseif option=B then . . .elseif option=C then . . . ” and so on. Menu activities function in asimilar manner. Branching can also be based on whether the previousquestion was answered correctly or not. Accordingly, corrective feedbackactivities may be provided.

In some embodiments, if a learner has already mastered a concept thenits adaptive path is not repeated just because the concept isencountered again in some adaptive path. However, in some embodiments,the concept is encountered again because the instructional designerspecified the need for repetition. Branching can also be based onoverall performance of the learner on a certain sub-concept.

Optionally, when branches feed into a stage marker, the activities afterthe stage marker cannot begin until the learner has completed allbranches feeding into the stage marker. This is a property of a stagemarker.

In at least some embodiments, the instruction engine ensures completionof branches feeding into a stage marker by checking to see if theactivity, concept link or set preceding each branch is complete. In someembodiments, the instruction engine records whenever each branch hasbeen taken into the stage marker. When all branches are ready, the stagemarker is considered “reached”. This stage marker ID is recorded in atable as the highest stage marker reached for that concept.

In another embodiment, the branching approach is probability-basedbranching, which enables assigning a certain relative probability thateach branch be taken. At runtime, the instruction engine essentiallyrolls dice and picks the highest number.

Branches can be optional or required. In at least some embodiments, theinstruction engine keeps track of this so that only required activitiesare automatically initiated. Optional activities are initiated by thelearner. And, if a learner chooses to do an optional concept link thatleads to required branches, those branches are still considered optionalsince the entire concept is optional.

Several branches can emanate from a single icon in an adaptive path. Ifthese branches emanate from a black circle nib or if they emanate frommore than one nib, then multiple activities or icons may becomeavailable to the learner. On the other hand, if the branches all emanatefrom a single diamond nib, then only one branch can be taken—the firstone whose branching condition is true.

Relating to a delay range, a minimum delay and a maximum delay areincluded in the branching criteria of each branch. When an icon is addedto the list of available activities, the date of availability isrecorded as well. Specifically recorded are: (i) the absolute date andtime that the icon will first become available, and (ii) the absolutedate and time that the icon will become urgent, meaning the max delayhas been exceeded. These are calculated by adding the min and max delaysto the current date and time at the moment that the branching conditionswere evaluated.

In generating a to-do list that presents to the learner a few, highpriority learning choices based upon the learner's previous performanceand upon learning sequences and priorities set, analyzed, and updated bythe system, the instruction engine performs a lot of behind-the-sceneswork to make the learner guidance system function. The learner guidancesystem is mostly a user interface, whereas the instruction engine isprimarily underlying intelligence that drives the user interface. Thisinformation is accessible via special activity components that supplythis information to the learner guidance system's user-interfacecomponents.

The to-do list communicates with the instruction engine in order to getthe items that are to be displayed. The instruction engine replies witha series of “MenuItem” objects, each of which hold information needed todisplay the object, as well as a series of objects that allow theinstruction engine to derive a series of available activities from it.For example, the to-do list may contain a “Test” row, which represents a“MenuItem” that contains every stage marker of type “Test”.

Relating to generating a progress bar list, the instruction engine knowshow to measure the learner's progress on every concept. This informationis then used by the learner guidance system to display a progress barfor each concept. Measuring depends on whether the adaptive path inquestion contains concept links. If the adaptive path contains conceptlinks, the instruction engine takes into account the extent to which thelinked concepts have been mastered and then adds all of the weightedpercentages to get a total percentage for the concept. (Concepts areweighted according to the estimated average time that it would take fora typical learner to complete all the activities in its adaptive path.)

If the adaptive path just contains activities, then the percentage ofprogress is equal to the percentage of activities completed, weighted bytheir completion time estimates.

A progress bar list component communicates with the instruction enginein order to determine what should be displayed in its list. Each row inthe progress bar list represents a MenuItem object, which holdsinformation needed to display the object, as well as a series of objectsthat allow the instruction engine to derive a series of availableactivities from it. For example, if the progress bar list shows theconcept “Banana”, then the MenuItem containing that particular conceptwould be passed to the instruction engine when selected by the user sothat the instruction engine could evaluate the adaptive path for thatconcept in order to derive one or more available activities.

In some embodiments, each MenuItem object provides a method that allowsit to tell other objects the text that should be used when displayingit. A MenuItem can hold any combination of one or more concept types,concepts, stage markers and activity instances. The MenuItem alsocontains a RecordUserData property, which indicates whether or not userdata resulting from use of the MenuItem should be recorded in thedatabase. Each time the instruction engine receives a new MenuItemobject it generates a new sub-available activities list, which has itsown current available activities list, previous available activitieslist, and working pool. Each sub-list determines whether or not userdata gets recorded, as determined by the corresponding MenuItemproperty. Each sub-list acts as a logical grouping of availableactivities. Once a sub-list has been exhausted the learner will bereturned to the previous sub-list. Once the learner has exhausted allsub-lists he or she will be returned to the progress bar list or to-dolist.

Regarding a representative process of choosing which activity to donext, the instruction engine sorts available activities at the moment anew available activity is added to the list, according to urgency date(first) and the distance of the available activity from the originallyselected concept (second)

Regarding a process of filtering by what the learner chose, afterlogging in the learner is presented with the menus of the learnerguidance system. These menus (e.g., a to-do list and progress bar list)allow the learner to choose a particular concept and optionally stage orconcept type to work on next. The learner picks a concept type from theto-do list. Before deciding which activity to do next, the instructionengine honors the learner's request by ruling out activities that arenot related to what the learner just chose.

To be considered ‘related’, the candidate activity comes directly fromthe selected concept's adaptive path or indirectly from one of itsconcept links any number of levels away. If the learner chose aparticular stage, then the activities or concept links are within thatstage or are descendants of concept links within that stage.

In at least some embodiments, preference is given to activities comingdirectly from the adaptive path of the concept originally selected bythe learner. In other words, activities that are related through one ormore levels of concept links are less favored.

Regarding immediately available activities, the first step for theinstruction engine in choosing which activity to do next is to see ifthere are any activities or icons that just became available as a resultof the preceding activity with a delay range of 0-0 (meaning the min andmax delay are both set to “immediately”). These activities are added tothe top of the available activities list in the order in which they wereencountered.

For activities where the urgency date is in the past, the working poolis filled if possible and the instruction engine proceeds with theactivity even if it can't be filled. As provided herein, term “workingpool” refers to a set of concepts being learned together, similar to theflash cards approach to memorizing multiplication tables, where thelearner works with a small set of flash cards at a time instead of theentire stack. In one embodiment, as soon as an activity is selected tobe launched by the instruction engine, if a working pool doesn't existyet, the instruction engine attempts to create one. If the instructionengine fails to create a working pool, and there are no activitiesremaining with sufficient working pools that are related to thelearner's original choice from the to-do list or progress bar list, andthe learner hasn't had a chance to spend much time on the selectedoption, then the instruction engine asks the learner whether to continueanyway, even though the working pool is incomplete.

The working pool continues until the instruction engine can no longerrefill the working pool and no urgent concept remains. The instructionengine discontinues the working pool if the learner has spent enoughtime with the current activity (e.g., the ideal concepts before activitytransition value has been reached). As soon as the working pool has beendiscontinued, the instruction engine looks for the next availableactivity.

A working pool includes several concepts with the same activity. In atleast some embodiments, concepts are kept in the working pool until thebranching conditions (subsequent to the activity) no longer requiresrepetition of the activity. In other words, if the same activityinstance is available, then the concept remains in the working pool.Otherwise, the concept leaves the working pool, which creates a vacancy.This is true even if a different instance of the same activity becomesavailable immediately for a given concept.

In some embodiments, the working pool is shuffled (randomlyre-sequenced) after every concept in the pool is seen (meaning theactivity is invoked for that concept). This auto-shuffling behavior onlyoccurs if the activities AutoShuffle property is set to True. Otherwise,shuffling only occurs when the navigation controller is told to shuffle,such as via a shuffle button. (The navigation controller is a specialactivity component that allows the activity to communicate with theinstruction engine.) Right before reshuffling, the instruction enginetries to fill any vacancies in the working pool with additional conceptsthat are ready for review with the current activity.

The number of concepts that should be in the working pool can be definedfor a given activity in an adaptive path. The default working pool sizeis defined at the activity level but the activity instance in a givenadaptive path can override this setting. The instruction engine findsall available concepts that are ready for that activity with the sameworking pool size. If there are enough to fill out a working pool, itwill start the activity. As concepts exit the working pool, theinstruction engine continues to refill the working pool as long as moreconcepts are available for the current activity. In one embodiment, whenone activity leads to several other activities, and all branches couldbe taken immediately, the sequence of the subsequent activities isintentionally randomized by the instruction engine.

At runtime, when a component or an activity fails the instruction enginemoves on to the next activity. The error causing the failure is loggedand uploaded to the central server. In some cases, it is helpful to letthe user know about the error, but in other cases it is better to justmove on to the next activity without the user knowing.

While waiting for user input on the current activity, the instructionengine pre-loads anticipated activities in the background for the sakeof speed. When an activity gets pre-loaded it's possible that it couldmake use of concepts that are not as relevant at the moment the activityis given to the learner. As such, in cases where an available activitymakes use of one or more related concepts just before the activity isgiven to the learner, the related concepts for that available activityare compared against the concepts in the working pool. If it is foundthat more than two-thirds of the related concepts are different fromwhat is found in the working pool then the activity is re-loaded inorder to make use of the concepts in the working pool.

In at least some embodiments, backward branching is allowed. This is auseful way to ensure long-term retention. Even after reaching themastery stage marker, a backward branching condition may provideperiodic review for that concept at increasing intervals. Specifically,the variable “PredictedDaysofRetention” can be included in theexpression that establishes a delay. This would apply to a givenconcept, and could be based on the longest period between reviews wherethe learner has been able to remember a concept.

In some embodiments, the instruction engine automatically keeps a backupof the user data. If user data is lost or corrupted, the instructionengine detects this and automatically restores the user data from thebackup. This backup is stored, for example on the local hard drive or onanother storage device. In some embodiments, several backups of the userdata are kept to minimize the risk of data loss.

As part of designing an activity, the instructional designer designateswhat the learner should do to complete it. For example, the learnermight have to remain on the activity for a certain minimum number ofseconds, click certain buttons, or fill certain text boxes. Toaccomplish this, the instructional designer sets the NextActivityEnabledproperty of the navigation object to be equal to the values ofproperties of other components in the activity.

While the learner is doing an activity, global navigation buttons allowthe learner to override branching. Accordingly, the learner can skip tothe next activity at any time or go back to a previous activity, jump tothe concept explanation activity (or any activity), return to the mainmenu to work on something else, or exit the program. In some activities,the learner can click on a next concept button to advance to the nextconcept in the working pool.

In at least some embodiments, the learner can defer items in the to-dolist. Deferred items appear at the bottom of the to-do list. The learnercan click on deferred items and do them any time, even though they aremarked deferred. In one embodiment, once the deferment date is passed,the item is no longer considered deferred. Instead, it shows up asnormal in the to-do list.

In at least some embodiments, optional activities are provided. Thelearner is given the choice of whether to do optional activities. If anoptional branch is encountered while the learner is progressing along anadaptive path, the instruction engine asks the learner whether he/shewould like to do the activity or not.

In accordance with embodiments of the present invention, the designermay change the instruction even after it has been distributed tolearners for use. The instruction engine identifies whether this hasoccurred. Thus, for example, when a designer deletes an activityinstance or concept any related available activities are deleted aswell. When a concept gets moved in the concept hierarchy, this movementhas no immediate effect on the instruction engine. However, the nexttime the learner encounters that concept will be different from the lasttime it was encountered by the user. If an activity instance or conceptlink is added to an adaptive path then the learner would simplyencounter that activity instance as he/she normally would, assuming theactivity instance is not added to a location the learner has alreadypassed. If the criteria for an activity set is changed then the newcriteria is used the next time the activity set is encountered.

In at least some embodiments of the present invention, the instructionengine interacts with a positive feedback generator to produce positivefeedback after activities, according to certain criteria, as will befurther discussed below.

Thus, as provided above, the instruction engine keeps track of thelearner's progress in learning concepts and initiates the instructionalactivities that are needed to further the learning progress. In someembodiments, when an activity becomes available to the learner accordingto the branching of an adaptive path, the activity is added to the listof all available activities for that learner. In some embodiments, aproperty of the activity may be set to someone other than the learner(e.g., a tutor, an instructor, etc.). Accordingly, activities or theresults thereof are made available to another individual or system, suchas a tutor, instructor, manager, proctor, etc.

B. Learner Guidance System

A learner guidance system helps the learner and tutor to develop apersonal learning plan for the learner, reports the learners' progress,guides the learner in what to focus on next, lets learners schedule taskpractices with tutors and peers, allows learners to set personalpreferences, and provides links to training regarding how to use thesystem and learn effectively. The learner also receives an automaticallygenerated report depicting his/her own level of implementation fidelityto a variety of features of the program. The learner may receivesuggestions or links to on-line training on how to improve. At least insome embodiments of the present invention, the learner guidance systemmay be built via activity builder and adaptive path builder. Withreference to FIGS. 26-27, a representative user interface in associationwith a learner guidance system is illustrated.

In at least some embodiments, a guidance system is divided into twoareas: (i) a learner guidance system (“LGS”) and (ii) a tutor guidancesystem (“TGS”). Both guidance systems typically have a to-do list tofacilitate easy access to high priority items. The following describesthe representative appearance and functionality of both to-do lists.

The LGS to-do list is divided into sections or panes representing thethree main types of activities a learner has access to: individual,collaborative, and optional. Relating to the individual, the requiredpane includes the following types of activities: test, review, new, andoptional activities that have been started. Everything a learner needsto be tested on is grouped by concept type and displayed on the to-dolist under the heading “Test.” The concept type with the greatestaverage urgency date is tested first, and so on. All material that needsto be reviewed is listed on the to-do list separately by concept type(e.g., phrases review, vocabulary review). The concept type with thehighest average urgency date has the highest priority, and so on.

New material may be displayed by task name or function. (e.g., MeetSomeone New [task], Aphabet [function]) In cases where an item isneither a task nor function, the item's concept type followed by eitherits concept name or another name is displayed. (e.g., Grammar: Modals,Pronunciation: Short Vowels) Material that a learner selects from anoptional pane will also be displayed with new material. New materialthat is most urgent has highest priority, and so forth. Items listed inthe first pane (individual) also have a designated completion date. Thisdate is found under the “Complete by:” column heading.

The collaborative pane includes all collaborative events, such as peerpractices and simulations, which are scheduled or need to be scheduled.This pane is prioritized according to the following hierarchy: (i) itemsscheduled for today, (ii) unscheduled items, (iii) auto scheduled items,and (iv) items scheduled for future dates. All events are displayedaccording to their task name, such as “Introduce Yourself and Others:Simulation.” A header makes the user aware of the amount ofcollaborative activities that fall into each of the categories listedabove. The following header is displayed above the collaborative pane:“You have x scheduled events.” “You have y unscheduled events.” “Youhave z events being auto scheduled.” Items in the collaborative pane arealso organized by when the event takes place.

A link to optional activities is displayed in the third pane of theto-do list. The link has a status bar which displays the number ofactivities a learner has started, deferred, and not started. When thelink is selected, the learner is taken to a separate list of optionalactivities. In at least some embodiments, the optional activities areprioritized according to the following hierarchy: (i) not started, (ii)items that are most recently available are displayed first (today beforefuture), and (iii) deferred, started.

C. Learning Optimizer

In accordance with at least some of the embodiments of the presentinvention, a learning optimizer tool is associated with implementationmodule 84 and/or analysis module 86. The learning optimizer will bediscussed below in association with analysis module 86.

D. Collaborative Activity Manager

A collaborative activity manager tool manages the initiation ofcollaborative activities by sending invitations to candidates who meetcertain criteria. In at least some embodiments, there are three mainmethods used by the collaborative activity manager to arrangecollaborative activities among multiple users. The instructionaldesigner assigns one of these methods to each collaborative activityhe/she creates. The first method is used when the instructional designerwants two or more learners to join in an activity once they have allreached that activity in their adaptive paths. In this case, thecollaborative activity manager keeps a list of all the learners thathave reached that activity, in the order they are received. Once thecollaborative activity manager finds a sufficient number of availablelearners to do the activity, who meet the criteria given by theinstructional designer, the collaborative activity manager invites eachof them to join. The learners receive instant pop-up messages invitingthem to join. If any decline, then the manager continues to the nextavailable person who is also waiting for this activity. This continuesuntil all the invitations are accepted, at which point the instructionengine starts the collaborative activity and then records the completionof the activity. The collaborative activity manager then removes theparticipants from the queue for that activity.

The second method of arranging collaborative activities is used in caseswhere a learner reaches a collaborative activity that can be done withother learners or tutors, regardless of whether the others have reachedthe same activity. In this case, the collaborative activity managermakes an immediate search for available participants, who meet thecriteria given by the instructional designer for each role in theactivity. The collaborative activity manager compiles a list ofresulting candidates, ordered in descending order of preference,according to additional sorting criteria given by the instructionaldesigner. Note that both the filtering criteria and sorting criteria areexpressions that may refer to any combination of properties of thepotential users, such as level of achievement, experience as a tutor,age, native language, background, sponsoring institution, learning team,and so on. These can also be weighted in an arithmetic expression byscaling any of the factors with a numeric constant. All of these are setby the instructional designer when creating the collaborative activity.For collaborative activities employing this second method, theinstructional designer may also specify whether the initiating learnershould have the option of selecting the other participants from a listof qualified candidates. In this case, the collaborative activitymanager presents the initiating learner with the names of the topcandidates (the instructional designer prescribes the maximum number ofnames to display). Otherwise, the top candidates are automaticallychosen by the collaborative activity manager. In either case,invitations are sent to the participants. If any decline, then thenext-most preferred candidates are invited, and this process continuesuntil the activity can start.

The third method of arranging a collaborative activity allows theparticipants to schedule a time for the activity in advance. In somecases, this may be preferred over the first and second methods, if thepotential participants are not often online using the software at thesame time. In this case, the collaborative activity manager provides theinitiating participant with a view of the schedules of other potentialparticipants, limited by filtering criteria set by the instructionaldesigner. The initiating participant may then send invitation(s) to joinin the activity at a certain date and time. Upon receiving theinvitation, the invitees may respond by accepting, declining, orproposing a new date and time. The invitee may optionally type a messageto be sent along with the reply. The initiating participant receives thereplies and can reply in similar fashion. This process continues until adate and time for the activity has been set. Reminders are sentautomatically by the collaborative activity manager before theengagement. The collaborative activity manager sends a finalconfirmation just before the activity is to begin. Participants mayaccept, decline, or defer for a few minutes. If any decline, the othersare informed, so that the activity can be rescheduled. If any defer, allare informed of the need to wait, and a count-down timer is displayed.Otherwise, the instruction engine starts the activity. As with otheractivities, all the user interaction can be tracked for collaborativeactivities.

E. User/Group Manager

In at least some embodiment, the system is used by many different typesof users. Accordingly, a representative group manager is used to placeindividual users into specialized groups in order to manage useraccounts more easily and efficiently. A user/group manager tool allowsusers, tutors, and administrators manage team structure or other groupswithin an organization. Specifically, the group manager tool is used tocreate, edit and delete user groups. It is also used to create newusers, define user information and then place users within an existinggroup. Group and individual user information, including pictures and allspecific properties of the group or individual, can be viewed using thegroup manager. It is also used to edit user information, as well asdisable, enable and delete users. With reference to FIG. 28, arepresentative user interface corresponding to a user manager tool isillustrated.

As provided above, embodiments of the present invention embrace othertypes of managers. For example, FIGS. 29-30 illustrate a user interfacecorresponding to an object manager tool. FIG. 31 illustrates arepresentative user interface corresponding to a group manager.

F. Positive Feedback Generator

As learners are presented with and respond to educational content, apositive feedback generator tool automatically provides encouragingfeedback to learners according to periodic reinforcement algorithmsoptimized for maximum motivation. The algorithms, in the simplest sense,comprise tallying correct answers for a particular amount of time andthen providing positive feedback when a particular number of correctanswers are reached. However, the algorithms and reward systems can bequite complex. For example, some representative circumstances of“payoffs” include: a learner does x things of type a correctly (orincreases their average number correct) a particular number of times ina row, in a session, or in a time period; a learner does x, y, and zthings correctly t times; a learner masters a particular body ofmaterial comprising a significant milestone in learning; etc.

Another area of variation is the ability to pull from pools of rewardgraduated according to desirability. This allows for meta-algorithmsthat could say, for example, if a learner meets the requirements foralgorithm a once, pull the reward from pool 1, if the learner meets therequirements for algorithm a again within t time (or t times in a row,or decreases their average time of fulfilling requirements by t time),then pull the reward from pool 2, etc.

Of course, the rewards themselves can vary widely and include anyvariation of text, graphic, sound, animation, movie, game, song, etc.divided into the various pools based on the degree of perceived rewardfor the learners.

G. Remote Document Feedback

The TALL software includes a set of components which can be combined tocreate a powerful document-creation, editing, and feedback system. Thissystem starts with a basic word processor (i.e., rich-text formatting)with spell-check capabilities. It allows learners to complete writingassignments and then send them to peers and tutors for feedback in asystematic way. Reference is now made to FIG. 32, which provides astoryboard of remote document feedback program as a representativeexample. As the author writes and the others review, criteria for boththe author and reviewers are made visible to encourage implementationfidelity. Receiving feedback is made possible by an email-like systemthat produces a list of peers and tutors that are qualified to providefeedback for a particular writing assignment and level. Once the draftis sent, peers or tutors are able to use a commenting functionality tomake suggestions for revisions. Comments are organized into categoriesto make the feedback easier for the learner to understand. The documentcan then be returned to the author or sent on to a more qualified peeror tutor. When the author receives feedback, the document systemdynamically creates and names new drafts. This system of drafts is meantto help the author use feedback to make improvements to the document.Once the author makes revisions, the document is sent back to peers andtutor for further feedback. They can comment on subsequent drafts andhelp the author continue refining the writing until it meets necessarystandards.

H. Rights Management System

A rights management system tool manages the availability of tools andfeatures in tools to specific users. FIG. 33 illustrates arepresentative user interface that corresponds to a rights manager tool.

I. Reports Builder

A reports builder tool enables the instructional designer to designreports to be included in the reporting system as well as in the course.The reports builder tool defines reports, questionnaires, tests, andother items about which data is gathered in the database and reported inthe reporting system. The data is collected from several differentsources. Such sources include via software application(s), observation,another tool of the system, questionnaires, examinations, etc.

With reference now to FIG. 34, a block diagram is illustrated thatprovides a representative association between a builder and a reportingsystem. In FIG. 34, information is provided from builder 90 in a varietyof manners. For example, the information may pass through a softwaretracking procedure 92, a test engine 93, an observation tool 94, or besent directly (as represented by arrow 95) to a storage device 96. Theinformation preserved in the storage device 96 is accessible by areporting system 98, which selectively manipulates or otherwiseprocesses the information and provides reports.

In one embodiment, the main screen of the report builder is divided intotwo sections: the “outline” on the left-hand side of the screen, and“edit” on the right-hand side of the screen. The outline screen isorganized into two major categories: (i) critical features and (ii)outcomes. Within each of these major categories lie two sub-categories:(a) components and (b) questions/items. Creating components is optionalfor both features and outcomes, however users cannot mix questions andcomponents on the same level.

The following options are available at all levels (e.g., outcome,feature, component, and question): (i) choosing a complete name, (ii)choosing a short name, (iii) defining a frequency, (iv) defining agroup, and (v) choosing whether to enable or disable a particularattribute. If the user changes the frequency of an item, all subsequentitems underneath that item will change accordingly (unless they aredirectly assigned an individual frequency). For software tracking only,the default may be daily and the options are daily, weekly, monthly, andannually. All other levels are user-defined based on the “frequencysentence.” Users have the option to (1) choose a frequency that isalready created from a drop down list box or (2) define a new frequency.

The default group(s) is based on the group of the direct parent item. Ifthe user changes the group(s) for an item, all subsequent itemsunderneath that item will change accordingly (unless they are directlyassigned an individual group(s)). Users have the option to (1) choose agroup(s) from a drop down list box or (2) define a new group(s).

The following options are available at the question level: (i) definingthe default chart type, (ii) indicating the units, range, and axislabels, (iii) establishing the minimum and ideal standards, (iv)choosing a start date, and (v) establishing accessibility. The defaultchart type can be one of various chart formats, such as pie chart, barchart, line graph, table, etc. In one embodiment, all items under thisquestion are set to this chart type by default. The units may be hours,minutes, number of concepts, and the like. The range controls the X andY bounds of the chart. The axis labels indicate what is being reported.Typically, the Y axis label indicates the measurement, while the X axislabel indicates the group and/or time period involved. The minimum andideal standards are displayed on the chart as colored lines or regions,so that performance data can be easily compared against standards. Thestart date indicates when the questions will start to be sent to thelearners, tutors, or others. The accessibility defines who shouldreceive the questions, based on criteria.

The following options are available at the item level: (i) choosing aname (optional) and (ii) choosing a source of datagathering—tests/questionnaires, an observation tool, using an automaticsoftware tracking of user data, and other sources of data.

With reference now to FIGS. 35-43, a representative user interface isillustrated that corresponds to a reports builder tool. In FIG. 35, arepresentative user interface having an edit screen associated with areports builder tool is illustrated. In FIG. 36, a frequency screen of areports builder tool is illustrated that is displayed upon selecting acategory of learner (e.g., a student that is learning a language) inFIG. 35 to bring up the “events” options illustrated in FIG. 36. FIG. 37illustrates a frequency screen brought up upon selecting “teacher” inFIG. 35. FIG. 38 illustrates a frequency screen brought up uponselecting “proctor-enabled” in FIG. 35. FIGS. 39-40 illustrate arepresentative user interface relating to a group screen in associationwith a reports builder tool. FIG. 41 illustrates a representative userinterface in association with an edit screen relating to a reportsbuilder tool, wherein the edit screen is on a question level. FIG. 42illustrates a representative user interface in association with an editscreen relating to a reports builder tool, wherein the edit screen is onan item level. FIG. 43 illustrates a representative user interface inassociation with a reports builder tool.

J. Research Organizer

In accordance with at least some of the embodiments of the presentinvention, a research organizer tool is associated with implementationmodule 84 and/or analysis module 86. The research organizer will bediscussed below in association with analysis module 86.

K. Tutor Guidance System (TGS)

The tutor guidance system (TGS) is designed to guide tutors through thetask of effectively tutoring learners. The following features of the TGSassist tutors in this task: to-do list, learner profile and reports,tutor notes, tutor session scheduling, and tutor training. Like theto-do list in the learner guidance system, this provides the tutor witha prioritized list of items that s/he needs to accomplish. The learnerprofile and reports provide the tutors access to information about eachof the learners they work with, including learner biographicalinformation, such as name, picture, native language, date startedlearning in system, and personal statement. It also includes reports onthe learners' performance during past learning sessions as well as goalsthat the learners have set to accomplish during future learningsessions. The tutor notes section allows the tutor to enter notes abouteach learner's progress and share these notes with other tutors. Thetutor also receives an automatically generated report depicting his/herown level of implementation fidelity to a variety of features of theprogram. The tutor may receive suggestions or links to on-line trainingon how to improve. The tutor session scheduling allows the tutors toview and, alternatively, set and edit times for tutoring sessions withstudents. The tutor training feature lists all the training the tutorneeds to be certified at various levels, allows them to choose to begina particular training module, and allows them to access completedtraining modules for review. It also provides access to embeddedcertification assessments. A representative embodiment is illustrated inFIGS. 44-45, wherein FIG. 44 illustrates student goals and FIG. 45illustrates a report generation.

L. Updater

In accordance with at least some embodiments of the present invention,an updater tool is utilized to keep information current.

M. Administrator Guidance System (AGS)

The administrator guidance system (AGS) is designed to guide localon-site administrators through the task of effectively administering thelearning system at a particular location. The following features of theAGS assist administrators in this task: to-do list, tutor profile andreports, learner summary reports, tutor scheduling, and administratortraining. Like the to-do list in the learner guidance system, thisprovides the administrator with a prioritized list of items that s/heneeds to accomplish. The tutor profile and reports provide theadministrator access to information about each of the tutors they workwith, including levels of implementation fidelity in a variety of areas,and tutor biographical information, such as name, picture, nativelanguage, date started tutoring in system, and personal statement. Italso includes reports on each of the tutor's performance during pasttutoring sessions, a list of a tutor's current and upcomingcertifications as well as goals that the tutor has set to accomplishduring future tutoring sessions. The learner summary reports providedata to the administrator on the progress of the learners at aparticular site and/or of a particular grouping as well as comparativedata to other, similar learners in other sites and/or groupings. Thisdata may include pre-testing outcomes that may determine whether aparticular person qualifies as a learner for a particular set ofcontent, and what part of the content they may need to study. Tutorscheduling allows the administrator to view and, alternatively, set andedit tutoring schedules for a group of tutors. The administratortraining feature lists all the training the administrator needs to becertified at various levels, allows them to choose to begin a particulartraining module, and allows them to access completed training modulesfor review. It also provides access to embedded certificationassessments.

The preceding discussion relates to representative tools that may beutilized to implement educational content in a dynamic continualimprovement educational environment. In at least some embodiments of thepresent invention, implementation of the educational content includescustomizably and dynamically instructing individual students/learnersand selectively gathering data relating to their learning theeducational content.

III. IMPLEMENTATION FIDELITY

Implementation fidelity refers to the degree in which the implementationof a given program is in harmony with the intended design. This refersnot only to the computerized portions of the programs, but also thedegree to which administrators, teachers and tutors, and students followthe intended design. In accordance with embodiments of the presentinvention, input from administrators, designers, or researchers candetermine what data is tracked, gathered, and reported. For example, inone embodiment, the tutor's computer-mediated interactions with thelearner are automatically tracked and reported to the tutor and anadministrator, who can then be aware of how closely the tutor is meetingfidelity standards. At the same time, the tutor can receive feedback orinstruction on how to improve fidelity, and/or the system can adapt toprovide the learner with additional interventions to make up for thefailure of the tutor to implement with fidelity.

The ability to track fidelity data, and the ability to improveimplementation fidelity significantly increases the ability to employcontinuous improvement.

IV. ANALYSIS TOOLS

In accordance with embodiments of the present invention, animplementation module also interfaces with an analysis module, which isemployed to evaluate the learning. The analysis module includes varioustools (e.g., learning optimizer, research organizer, etc.) to performthe evaluation. Based on the evaluation, a modification module may beemployed to selectively customize educational content, a frequency inwhich content is presented, an order of content presentation, or anyother factor to customize the teachings to the individual user, group orlesson. Accordingly, embodiments of the present invention embrace theutilization of a variety of tools that enable the analysis of teachingwithin the dynamic and customizable continual improvement educationalprocess. Representative examples of such tools include a learningoptimizer and a research organizer, a discussion of which will bediscussed below.

A. Learning Optimizer

A learning optimizer tool allows instructional designers to experimentwith various options, in order to tune the system for optimal learningefficiency. Results are analyzed automatically, and the systemrecommends and in at least some embodiments even automatically adjustsvariables to their optimum settings. The learning optimizer is in shortthe brain behind the research organizer. It performs analyticalfunctions based on inputs defined in the research organizer.

There are four types of inputs or variables defined in the researchorganizer that are needed in order for the learning optimizer to performits function. The variables describe (i) the learner, (ii) theinstructional system, (iii) the situation, and (iv) responsemeasurements. Learner variables describe the person using the system,whereas learning history reports the choices made by the learner.Instructional factors describe properties of an activity or adaptivepath that are set by the system or determined by developer. Thesefactors are on the learner or concept level. The user has no controlover the frequency or levels of these factors. Situational factorsdescribe the surroundings and environmental conditions of the learner.Response measurements are variables to measure how well the user islearning his/her target subject (e.g., language or other educationaltopic).

The applicable variable list is found in several locations throughoutthe system including the instructional builder tools (activity builder,adaptive path builder, etc) and research organizer.

For the purposes of the learning optimizer, the variables can also beconsidered in terms of three different categories, namely (i) fixedvariables, (ii) adjustable variables, and (iii) response outcomes. Thespecification of fixed variables and adjustable variables in theresearch organizer determines how the learning optimizer constructs theexperimental design. Fixed variables most commonly describe the learner,while adjustable variables most commonly describe the properties of theparticular instructions system. Adjustable factors and response outcomesconstitute the variables signifying the experimental intervention andmeasurement of its success. Situational variables are included as bothfixed and adjustable variables.

The learning optimizer identifies and optimizes the relationshipsbetween these sets of variables using information obtained from theresearch organizer. The research organizer has the user (instructionaldesigner or researcher) select the adjustable variables that are to beexperimented. These variables are created in the object manager andappear for use in the Research Organizer. (An exception to both of thesemight include an experiment designed to test situational factors.) Theprocess also requires the researcher to specify which response outcomescorrespond with the chosen set of adjustable variables involved in theexperiment.

The experimenter can control for (take into account during the analysis)fixed variables by specifying them during the proposal process. Fixedvariables that are categorical (e.g., days of the week or gender) arecalled blocking variables. Fixed variables that are continuous in nature(e.g., age, education level, ESL background) are called covariates. Thisinformation is optional. For very specific learning contexts it is notas important to specify blocking variables or covariates, however forbroad learning contexts it is recommended as homogeneity decreases.Specifying the adjustable variables, the fixed variables and theresponse outcomes allows the learning optimizer to create the correctdesign.

Four distinct items or sections are completed when proposing a newstudy, which directly relates to the creation of the experimentaldesign. They will each be addressed in order to understand how and whythe learning optimizer chooses a particular design for an experiment:

1. Define Learning Context

The purpose of the defining learning context is that it determines whois included or excluded from the study. An example of a learning contextcould be: all students with at least one year of a particular experience(e.g., training). This excludes anyone with less than one year of theexperience regardless his or her race, education level, learning style,gender, age or any other important characteristic.

2. Define Instructional Factors

The purpose for defining instructional factors is to determine whetherthe experiment is a treatment comparison or factorial design, whereinone set of treatments is tested over one or more other sets oftreatments.

In at least some embodiments, the learning optimizer builds anexperimental design based solely on which adjustable factors are linkedto the experiment and for which fixed factors are specified as blockingvariables or covariates. Response outcome variables are not necessaryinformation for this step in the process.

There are three types of experiments that the learning optimizer canperform, each with several design choices. They are: (i) treatmentcomparisons, (ii) screening factorial experiments, and (iii) responsesurface studies.

The learning optimizer recognizes the experiment as a treatmentcomparison study if a researcher assigns one variable with severalvalues to the experiment. In other words, the researcher selects onevariable with several realizations called treatments. Initially, allparameters governing each level are set to default values. Unlessblocking variables are defined, the learning optimizer randomly assignseach level (treatment) to one individual such that all treatments arerepeated the same number of times. This achieves balance and simplifiesthe analysis. The learning context provides the pool of possiblesubjects for the experiments. However, the learning optimizer and/orresearcher may determine that only a subset of the available learners isneeded (i.e. a subset of the total number from the learning context.)

The purpose of a factorial experiment is to choose optimal settings ofmany factors by taking into account the way they interact with eachother in the learning process. Each person receives a combination of thefactor levels (one level from each variable). One example is todetermine if certain types of activities are superior when paired withspecific other activities (and/or any combinations). This could bedifferent activities within one group of concepts, or differentactivities across groups. Performing a factorial experiment like this isuseful for detecting which combination of activities maximizes learning.Another example would be to perform an experiment within one activity soas to uncover what property settings should be used to maximize themastery of a particular group of concepts. Assigning several two-levelfactors to one experiment results in a factorial design, despite thenature of the factors. Because factorial designs are in part based uponthe number of available subjects, the learning optimizer searches forstudents in the learning context to determine the exact design that mustbe used.

3. Choose Experimental Unit

An experimental unit is defined as the basic unit that receives thetreatment or experimental intervention. Another way to determine whatthis unit is for a given experiment is to answer the following question:To what or to whom am I randomly assigning the treatments? Theresearcher is given the choice between having a person or “learner” asthe experimental unit and having a “concept” (e.g., word, phrase,grammar principle, or even a certain type of activity) as theexperimental unit. If each learner receives a different treatment, thelearner is then the experimental unit. However, if each concept for alearner receives a different treatment, then the concept is theexperimental unit.

Using the concept as the experimental unit has the potential for greatlyincreasing the statistical power of the experiment, in cases where itmakes sense. For example, an experiment involving only ten subjectswould most likely not yield enough data from which to draw anyconclusions, if the experimental unit is the learner. But if theexperimental unit is the concept, and the ten subjects each learnone-thousand vocabulary concepts, each concept being randomly assignedto a few different treatments groups, then the likelihood of detectingthe impact of different treatments is much higher, assuming there is adifference to detect. There are further advantages to using the conceptas the experimental unit. Gathering data points for eachtreatment/student combination allows an interaction to be uncoveredbetween the student and each treatment. This interaction reveals whattreatments are best suited to every student, and subpopulations can bediscovered. These subpopulations are poorly understood prior toconducting the experiment.

4. Define Blocking Variables or Covariates

Blocks and covariates are variables that describe the experimental unitor the experimental unit's situation. If the researcher selects“learner” as the experimental unit, he/she is then able to specify blockvariables or covariates if desired. However, until there are variablesthat describe concepts, such as difficulty level, specifying blockvariables and covariates will not be an option for concept-unit designs.Treatment comparisons experiments will thus be completely randomized,and factorial experiments will be traditional.

In the case wherein the learner is the experimental unit, the learningoptimizer allows the researcher to specify up to two blocking variables(e.g., he/she can specify days of week AND gender as blockingvariables). However, it will allow the researcher to consider as manycovariates as desired.

A second purpose of the learning optimizer is to calculate and presentthe statistical power of the design for different numbers of experimentunits. Before the learning optimizer actually implements the design itwill allow the researcher to select the number of experimental unitsfrom a list of possibilities presented in the research organizer. Thelist is derived from the total number of students in the learningcontext by considering only those numbers that ensure a balancedexperimental design. The research organizer presents the maximumpossible number, along with lower numbers, which may enable theexperiment to be completed in less time. The researcher can make his/herselection from the presented values based on two criteria: First, foreach choice in the list the research organizer will display theprojected amount of time to complete the study based on that particularnumber. Second, it will present a power analysis for each of thechoices.

The purpose is to show the researcher the average effect size he/she canexpect to detect from his experimental interventions based on thecorresponding number (e.g., students or concepts). Effect size is theamount of effect that each variable has on the response measurement.

When a concept has been chosen as the experimental unit, the choice ismore important to the researcher. Time plays an important role indetermining how many concepts to include in the experiment. Doubling thenumber of concepts to be included in the experiment doubles the amountof time it takes to complete the experiment. The learning optimizerprojects the length of the experiment for each number based on empiricaltrends and past data.

A third purpose of the learning optimizer is to perform the experimentwith the users specified under the learning context by correctlyapplying the treatments and collecting the data. For automatedexperiments, this application of the treatments is performed by thelearning optimizer in concert with the instruction engine. In atreatment comparisons experiment, each learner is assigned one level ofthe factor for the course of the experiment. This factor level is thetreatment for that individual. In a factorial experiment, the learnerwill be assigned one level of each factor based on the plan. The onecombination signifies the “treatment” for that learner. By reviewing thedesigns shown in the first section, it becomes clear that for bothfactorial designs and treatment comparison designs, the term “treatment”simply means a unique row in the design. Blocks are not included in thedefinition of the treatment.

The learning optimizer knows the meaning of each treatment and appliesthe correct levels of each factor to the correct people defined in thelearning context or based upon blocking variables. In either instance,the term “treatment” is synonymous with a row in the design. Once thetreatments have been applied to the experimental units, the data must becollected and compiled in a standard way to make it ready for analysis.It is also available in text format so that future researchers orvisitors to the research organizer may download the data set to a filefor personal examination.

A fourth purpose of the learning optimizer is to analyze the dataaccording to the type of design and to suggest actions to be taken as aresult of the experiment. The learning optimizer analyzes the dataaccording to the type of design and other criteria as entered in theresearch organizer originally. The following are types of analyses thatcan be performed by the Learning Optimizer: (i) summary charts andgraphs, (ii) significance tests, (iii) presentation of estimatedeffects/coefficients, and (iv) main effects plots and interaction plots.

Despite strong conclusions that favor one treatment over another, or onefactor level over another, the learning optimizer does not enforce (makechanges to parameter values) any conclusions without outside researcherinstruction, at least in some embodiments. The learning optimizer simplypresents the results of the study for the researcher to view in theresearch organizer. Once he/she has viewed the automatic analysesresults, the data and also conducted any other additional analysis asdesired, the researcher has the option of what to do with theinformation. At least some embodiments of the present invention embraceautomated alteration of the instructional design based on the results.

Thus, a learning optimizer tool allows instructional designers toexperiment with various options, in order to tune the system for optimallearning efficiency. Results are analyzed automatically, and the systemrecommends or enforces optimum settings. It functions by submittingstudies in the GUI interface, namely a research organizer tool.

B. Research Organizer

A research organizer tool allows researchers and developers to see theresults of experiments, to suggest new experiments, to see data andrationale for making changes to instructions, and to see importantfindings highlighted. The research organizer is the center and userinterface to review old studies and propose new ones. FIG. 46illustrates a representative user interface in association with aresearch organizer tool.

The following is an explanation of three levels of meta-data. The firstlevel (“level 1”) of meta-data includes the three main summarydescriptors of every study: title, author(s), and start date. It isrequired before submitting a study. This information is chosen by theresearcher and input directly into the research organizer. The secondlevel (“level 2”) includes other keyword descriptors that apply to bothmanual and automatic studies, also input directly into the researchorganizer, but is not required to submit the study. The third level(“level 3”) is slightly different for automatic and manual studies. Forautomatic studies this information is input immediately after completingthe level 2 meta-data as the information will be used to create thedesign. For manual studies the level 3 meta-data is input once the studyhas been concluded and is simply for informational purposes. For bothtypes of studies, the information is input into the research organizeritself. For automatic studies this includes defining variables andchoosing the response. As explained previously, all variables arecreated in the object manager, selected in the research organizer tool,and implemented in the activity or adaptive path builder tools.Visually, in at least some embodiments there are three main sections tothe research organizer. They are: (i) the study finder, used to searchfor and select studies for viewing, (ii), the process support, theequivalent to a help feature, and (iii) the research study panel, usedto propose a new study or view a previously submitted study.

1. Study Finder

A study finder is used to search for and select studies for viewing.When on the index tab the researcher automatically views all studies inthe database. These studies are ordered by specific headings chosen bythe researcher. The studies are ordered by Title as a default, but canbe ordered in other manners.

The search tab allows a researcher to select a keyword category out ofthe combo box and enter the corresponding keyword in the text field. Theuser can enter up to 8 different keywords and connect them with AND, ORand NOT. Once the information is entered, the researcher hits SEARCH andthe system will immediately bring up the Results tab screen containingthe matches. Upon clicking back to the Search tab the keywordspreviously entered by the user will have automatically appeared in thesearch history box. The user can either modify existing information andSEARCH again, or he can RESET the Search screen. Resetting the Searchreturns all categories in the combo boxes to their original positionsand clears all keywords entered by the user. A completely new search canthen be entered. Note: every time the SEARCH button is clicked a newentry is added as a new search in the Search History box.

Initially this tab is disabled, but after hitting the SEARCH button, theresults appear by default. They are presented on the tab in a similarmanner as the index portion of the Study Finder presents all studies.Headings to sort by can be included and excluded similarly to theheadings on the Index. If the search was unsatisfactory or after gettingall the desired information from one search the researcher can then goback to the search screen, reset and try and new search.

2. Process Support

The process support is equivalent to a help feature. This featureincludes information on how to complete a task in the researchorganizer. The topics are provided in an attempt to guide the researcherthrough any of the research organizer's functions. Process support givesdefinitions, tips and instructions to accomplish different tasks.

The methods to find help on a specific topic are facilitated by thegeneral help system for all tools. Help for a research organizer toolcan be accessed from the tool menu or by pressing the F1 key when insidea research organizer tool. For quick help however, tool tips on eachlabel and table heading provide simple instructions and explanations.

3. Research Study Panel

The research study panel is used to propose a new study or view apreviously submitted study. In the present example, the research studyportion of the research organizer includes of six tabs: 1) Meta-data, 2)Proposal, 3) Human Subjects Review, 4) Data Set, 5) Results, 6)Analysis, and 7) Comments. The comments screen is available for alltypes of studies and is a forum to submit new comments and review pastcomments. Tabs 1-3 and 7 are available for completed studies, ongoingstudies, and studies scheduled in the future. Tabs 4-6 are available forstudies that have been completed. The summary statistics and graphsfound on the results tab are available for ongoing studies and areupdated weekly.

The meta-data tab includes text descriptors of the study, but noresults. The information is presented in order of the meta-data levels.Level 1 data as previously explained is shown first (includes the Title,author(s), and start date.) Level 2 information follows and includes thepurpose, the learning context, the location(s) of the study, therationale behind the study the other researchers involved and how theexperimental factors were defined (i.e. in the object manager or from anoutside source.) Finally the correct Level 3 information is presenteddepending on the type of study (automatic or manual). A more detailedexplanation of how this information is entered is found after thedescription of the seven different tabs.

The proposal tab presents a written proposal of the experiment. Forresearchers who are designing a NEW study, there are two options forentering the proposal. First, the screen includes a blank text box toallow the researcher to write a proposal directly on the tab or copy andpaste in a previously written proposal. Second, a text field to enterthe path and a browse button to locate a previously written proposalappears on the screen. This allows the researcher to upload a documentinto the research organizer instead of being required to write itdirectly into the screen. The human subjects review tab provides achecklist of those items that are to be completed before the study canbegin. This includes the application, literature review, proposal ofmethods and other documentation that is required in order to perform astudy with human subjects. The last item is a check box thatacknowledges receipt of an approval letter. Each of these items must bechecked off and the date of completion entered. Once the approval letterhas been received, and the meta-data and proposal sheets aresatisfactory to the researcher, he may submit his study to the learningoptimizer. There will be a button located on the meta-data screenindicating final submission. Once the study has been submitted, it willappear in the system as a future or current study in the study finderand can be viewed by all visitors to the research organizer. However, itwill not actually start until the start date specified by the researcherin the meta-data sheet.

Regarding the data set, if the study was performed manually the data areuploaded to the system in text format. This is achieved similarly touploading a proposal. The researcher enters in an explanation of allvariables (in the correct location which appears as a text box only inthe case of manual studies). This option is disabled when the researcherhas already uploaded the data, and also when an automatic study hasoccurred. At this point the data is already compiled and stored in thecorrect location automatically. The explanation of variables alsoappears by default. These explanations are based on how they werechosen, changed and created in the instructional builder.

In the future, any visitor to the research organizer can download thecomplete data set to a file for additional analysis or study through asimilar interface. The only options available on this screen forvisitors are the explanations of variables and the download data option.

Summary tables and graphs are presented first and are available as thestudy is ongoing. They are updated weekly until the study's conclusion.These are standard graphs that will automatically be built by thereports builder and linked to this location. After submitting anautomatic study the researcher will have the option to keep the standardgraphs or personalize the results to display graphs of his choice. Ifthe researcher finds the standard graphs unsatisfactory, he can enterthe reports builder and change what graphs are linked to the resultspage, i.e. personalizing his results page.

For treatment comparisons the automatically generated graphs include thefollowing: a histogram of each response variable for each treatment; anoverall bar chart with the response variable averages plotted for eachtreatment; and a “default use of time” style chart with averages foreach treatment. For a factorial design there will be a table of theresponse variables for each factor level across all factors, a bar chartcomparing overall means for each response variable (i.e. bar chart ofthe previous table), and finally histograms of each of the overallresponse variables may be presented. For each type of experiment, thegraphs will be presented by block representations if blocking variableshave been specified.

Each graph is accompanied by a standard explanation of the graph typeand how it should be interpreted. These explanations are viewed byclicking the button next to each graph. In conjunction with the summarystatistics and graphs is an abort button that allows the author(s) tostop an automatic study in the (rare) case it seems to be an absolutefailure. This button is disabled to all visitors that are not identifiedby the system as the author of the study. Once the study has beenaborted it is not possible to resume the same study. A completely newstudy must be proposed in order to examine the same things.

Next the statistical analysis is presented. The analysis performedautomatically includes: 1) graphs indicating the significance ofexperimental factors and 2) tables presenting results of hypothesistesting and estimated effects (coefficients). General information aboutinterpreting the plots is located in the help feature.

Regarding manual analysis (referring to the analysis tab rather than amanual study), analysis refers to critical thinking and conclusionsdrawn by humans in contrast to results, which refers to proceduresconducted automatically by the computer. The information on this tab maybe input by the researcher regardless if the study is automatic ormanual. The analysis tab may include a text box that suggests ideas ofinformation to be entered, although it is ultimately up to theresearcher. For example, the following information may be suggested:What additional statistical analyses did you conduct and what did youfind? What theories do the results support of suggest? What changes ifany to the system will result (document the before and after)?

Once all the text fields are filled, the researcher can electronicallysubmit the information and it will be stored with the study as publicinformation to be viewed by other researchers.

The comments tab is available when viewing any study off the studyfinder, meaning a future scheduled study, an ongoing study or acompleted study. The majority of the screen is a text box that allowsany researcher to type in comments and submit them to the system. Theperson submitting comments must also enter a subject in a separate textfield, similar to an email. The system will categorize all comments bythe subject, author (identified by user number when logged into thesystem) and date of submission.

If the user is recognized by the system as one of the authors, a buttonwill also appear enabled that will bring up comments previouslysubmitted about the study. Comments are accessed by inputting thepassword (chosen by the author at time of submission). Once the passwordhas been correctly input, a new screen will appear that displays thecomments. Comments are presented similarly to how studies are presentedin the Study Finder. They will be listed by the headings of author,subject and date. The researcher can sort them in ascending ordescending order according to one of the headings by clicking on it.

The research creates a new study by clicking on the “Add Study” button.After the title is input into a text field, the meta data sheet askingfor with all required information will appear. The authors are chosenfrom a combo box one at a time and added to a list. There is also afeature to add a researcher to the combo box if the desired person isnot already listed. Clicking on the Add Author button will bring up abox that has a text field wherein the user types the name to be added tothe list. The system will not only add it to the combo box of names forfuture users, but include it in the current list of authors for thatstudy. Finally the user will be required to type in the suggested startdate into a text field in the following format: mm/dd/yyyy.

Level 2 information is also required information for all types ofstudies (automatic and manual). The first entry in Level 2 asks for thepurpose of the study, which is typed directly into a text field. Thelearning context is the second entry in Level 2, and is definedsimilarly as in other parts of the system such as the activity builder.

The researcher is required to decide if his experimental factors can bedefined using other tools. In at least some embodiments, only thoseexperiments whose factors can be defined by the tools will have furtherinformation that must be defined in the meta-data sheet beforesubmitting the study, i.e. Level 3 meta-data. If the researcher clicks“NO” to defining the factors using the tools, then a pop-up box appearswhich reminds him of the information that he must enter into the systemfollowing the conclusion of his study. The information includes:completing the Level-3 meta-data for manual studies, uploading the dataset, and inputting the results and analysis. Another box contains twotext fields to select (and re-enter) a password for the study. Thepassword allows the author to view comments about his study in thefuture. He is now free to exit the Research Organizer and begin hisstudy.

Once the researcher has selected “Yes” to defining the factors in thecore tools, he must first select his experimental unit for the study:learner or concept. A follow up question asks him to decide whether hewould like to use current TALL learners, future TALL learners, or acombination. Choosing “new” (current) learners will include a specificnumber of users as they enter the system until the quota is filled.

The experimental variables are selected from a combo box displayingthose previously created in the object manager. One variable may beselected from the variable combo box and additional variables can beadded through a list. Specifying further variables creates a factorialexperiment. Only after selecting a two level variable from the combo boxis the list of other two level variables enabled. Two questions areenabled if the study is a treatment comparison. They ask for a controltreatment and whether or not the author is interested in finding the“best” treatment. Control treatments and (a guess at) the best treatmentare chosen from a combo box that contains only those levels of theexperimental variable associated with the experiment. Only one item perquestion may be chosen and nothing can be added to the list.

The researcher can select the categorical (blocking) variables tocontrol for, and also the values that they take on from lists presentedin combo boxes. The researcher may not add to these lists as he haspreviously been permitted to do. For example, by choosing the variableday of the week, he must also choose the values he would like toinclude, such as Monday, Wednesday and Friday. Thus, each variable boxwill only allow the researcher to select one item, while the boxescorresponding to the values, will allow him to select multiple.

He is then asked to select from a list of continuous variables(covariates) for which he would like to control. These include thingssuch as age, education level, years or months of ESL experience andothers. If a variable cannot be broken down into a small number ofstandard and finite categories, then it is probably a continuousvariable. The researcher may select multiple values from the list, butcannot add new entries except by creating new variables in the objectmanager tool.

Finally the researcher chooses one or more response variables whichmeasure the success of his experimental treatments or factors. Examplesof these response variables include: percent correct, number mastered,learning efficiency. The researcher can create variables through acalculator type mechanism also seen inside the instructional builder.Defining the response is important so that the system will know whichinformation to compile together and analyze.

To complete the meta-data sheet, the researcher may select the totalnumber of subjects to include in his study. This may only be a guess ata first passing, but before the study is submitted the researcher shouldfeel confident in his selection. The system will present choices for thenumber of experimental subjects to include in the study based on thecriterion that he has previously entered. This criteria includes thetotal number of subjects available in the learning context, the type ofexperimental unit, response variable information, and if applicablewhether or not the experiment is using current or future users, or both.

The decision of how many subjects to include should be guided by theaccompanying projected amounts of time that the study will take to becompleted, along with the predicted detectable effect of the treatment.Each choice will be paired with a predicted length of time based uponincluding that particular number of subjects. The prediction isprimarily based upon historic or past data and also knowledge by theresearcher. This number is also based upon whether the researcher wouldlike to use current users, or limit the study to only future or “new”users.

Researchers should also select the number to include by viewing thestatistical power of including differing amounts of subjects in theexperiment. The greater the number of subjects, the greater power theexperiment will have to detect a significant effect from theexperimental interventions. Until the final plan is submitted, theresearcher can make any changes he desires to the meta-data information.Once the plan has been submitted however, the meta-data sheet becomes atab with text presented on it, and it cannot be changed or edited. Theproposal and human subjects review tabs have the same limitations.

Upon completing all tasks explained above the researcher will havefinished the meta-data sheet requirements. Upon submitting the finalplan (which only happens after writing a proposal and getting approvalfrom the Human Subjects Review), the system will design the experimentbased on the above information, and return the meta-data sheet in itsfinished text form as seen on page 1. The system will have also filledin the final two pieces of information presented in the meta-data sheeton page 1. These are the type of study and the type of design. Thisinformation can only be determined by the system after all priorinformation is gathered. These two additional descriptors of the studyare only for information and classification purposes and are not enteredor chosen directly by the researcher.

Once the study has been submitted, there a pop-up box appears tellingthe researcher what information he is required to input into the systemat the conclusion of the study. This primarily includes the “human”analysis, found on the analysis tab. There is also a place to choose apassword for the study that must be entered in the study is ever abortedor if the researcher would like to view comments about his study.

Another pop-up box allows the researcher to add information to one ofthe lists previously encountered. The box appears basically the same foreach category except that only the correct category is bolded, while therest appear disabled. This reminds the researcher what he is adding tothe system. If he decides not to add an item, he can click CANCEL toreturn to the meta-data sheet, or he can type in a new item and click OKto have it added to the list.

A pop up box appears if the researcher selects NO to defining theexperimental factors in the instructional builder tools. This simplylists the information that is required to be input into the system. Tosubmit the study the researcher must still press the SUBMIT button. Asimilar pop up box appears after clicking YES to define the factors inthe instructional builder. Significantly less information is required atthe conclusion of an automatic study. There are also two boxes that thesystem requires the researcher to pass through if he is to abort anautomatic study. They include a confirmation box and one requiring thepassword (and then a re-entering of the password for confirmationpurposes) chosen by the researcher.

Thus, in accordance with embodiments of the present invention, animplementation module also interfaces with an analysis module, which isemployed to evaluate the learning. The analysis module includes varioustools to perform the evaluation. Based on the evaluation, a modificationmodule may be employed to selectively customize educational content, afrequency in which content is presented, an order of contentpresentation, or any other factor to customize the teachings to theindividual user, group or lesson. Accordingly, embodiments of thepresent invention embrace the utilization of a variety of tools thatenable the analysis of teaching and learning within the dynamic andcustomizable continual improvement educational process. Automatedexperimentation is an important part of the overall method of thepresent invention, which makes the continual improvement educationalenvironment feasible. This method of automated experimentationaccelerates the improvement of education well beyond the rate oftraditional educational research methods.

V. OTHER TOOLS

Those skilled in the art will appreciate that other tools may beutilized in a dynamic continual improvement educational environment.Examples or such other tools include a data and code synchronizer andother tools.

A. Data and Code Synchronizer

A data and code synchronizer tool transfers both data and code between aclient computer and a central server in order to ensure that the systemremains up-to-date.

VI. CONCLUSION

Embodiments of the present invention take place in association with adynamic learning process that includes the ability to design or developan educational experience or otherwise provide educational lessons(e.g., educational activities, educational content, etc.). The designingof the educational experience or lesson is facilitated by utilizing anobject oriented format, a drag-and-drop interface, or other process thatfacilitates design development of educational content and does notrequire a programmer to develop the educational experience. Oncedesigned, the implementation of the educational lesson is experienced byan individual or group, and includes providing instruction and gatheringdata. An analysis is performed on the data to optimize learning. Theanalysis corresponds to a particular individual, group and/oreducational lesson. Modifications are selectively or automatically madeto the educational lesson. The process of designing, implementing,analyzing, and selectively modifying creates a cycle that optimizes thelearning process and adapts to groups and individual learners with thegoal of improving learning outcomes and efficiency.

At least some embodiments of the present invention embrace theutilization of a computer device in designing educational lessons,implementing educational lessons, performing analysis and/or providingmodifications. An individual learner/student may interface with thecomputer device in the educational environment and may additionallyinterface with an instructor. If this instructor (or a peer) is at adistant location, their collaboration may be mediated through multiplecomputers connected to the internet or some other network.

Within the environment, the educational lesson or content is dynamicallyprovided to the learner on an iterative basis according to the need ofthat learner in the learning process. Learner performance data isgathered and is selectively used to adjust the pace of learning, tomodify the frequency of exposure to particular content, and to regulatethe type and difficulty of content to which the learner is exposed.

Aspects of the educational environment are easily and/or automaticallyadapted to a learner's performance. For example, if an analysis of userdata indicates that a given learning activity needs an additionalfeature, that feature can quickly be added or created, such as throughthe use of a drag-and-drop interface. Similarly, if a given activityproves to be unhelpful, it can be immediately eliminated. Also, if aparticular activity proves to be helpful to some learners, but not toothers, entry conditions are set to only allow those learners that arepredicted to benefit from the activity to be exposed to the activity.Factors or characteristics of the user that may be taken into accountinclude age, native language, learning style, institution, background,interests, purpose in learning, degree of long-term retention desired,the breadth or depth of their overall mastery, and their need to beparticularly well-prepared to use a certain subset of the informationfor an upcoming responsibility or engagement (such as an academicconference on a particular subject), etc. Other learners skip or neverexperience the particular activity. Alternatively, they may be exposedto another educational activity. If a new activity would be useful tothe learner, the new activity may be quickly provided or otherwiseauthored by a designer rather than a programmer.

Automatic or partially automated studies determine the effectiveness ofparticular lessons, activities, or other instructional design decisions.For example, an assignment may be made for a first test group toexperience a first lesson and a second test group to experience a secondlesson. In another example, one learning activity might be presented ina different way to one group than another group. The results of the twotest groups are analyzed to determine the effectiveness of the twolessons in relation to each other.

While the methods and processes of the present invention have proven tobe particularly useful in the area of teaching a foreign language to thelearner, those skilled in the art will appreciate that the methods andprocesses of the present invention can be used in a variety of differentapplications and in a variety of different educational environments toteach any type of educational topic or material. For example theeducational content may embrace language, mathematics, science,technical training, cooking, medical procedures, a particular skill,professional training, or any other learning.

Thus, as discussed herein, the embodiments of the present inventionembrace providing a dynamic continual improvement educationalenvironment. In particular, the present invention relates to dynamicsystems and methods for gathering/tracking data, automatically adaptingto an individual's pace of learning or other characteristics of thelearner, selectively determining the type and difficulty of contentprovided to an individual, selectively providing an exposure frequencyfor the content, and/or enabling rapid design modifications within theeducational environment.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges that come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A method for providing a dynamic continualimprovement educational environment for a user, the method comprising:designing dynamic educational content for presentation to the user,wherein concepts of the educational content are graphically linked in arelational order; selectively implementing the presentation of theeducational content to the user, wherein the presentation isautomatically adapted to a learning pace of the user; and iterativelyimplementing at least a portion of the presentation to the user over anextended period of time to maintain the user's understanding of theeducational content.